Details
-
Type:
Improvement
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.7.1
-
Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
-
Component/s: documentation
-
Labels:None
-
Hadoop Flags:Reviewed
Description
Some Kerberos configuration parameters are not documented well enough.
Issue Links
- breaks
-
HADOOP-14229
hadoop.security.auth_to_local example is incorrect in the documentation
-
- Patch Available
-
- is duplicated by
-
HADOOP-12874
Configuration table is broken in HttpAuthentication.md
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 19m 22s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | javac | 8m 1s | There were no new javac warning messages. |
| +1 | javadoc | 10m 39s | There were no new javadoc warning messages. |
| +1 | release audit | 0m 25s | The applied patch does not increase the total number of release audit warnings. |
| +1 | site | 3m 32s | Site still builds. |
| +1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
| +1 | install | 1m 44s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 37s | The patch built with eclipse:eclipse. |
| +1 | common tests | 7m 27s | Tests passed in hadoop-common. |
| -1 | hdfs tests | 50m 51s | Tests failed in hadoop-hdfs. |
| 102m 41s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| hadoop.hdfs.qjournal.TestSecureNNWithQJM | |
| hadoop.hdfs.server.namenode.TestSecureNameNode | |
| hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12766986/HDFS-9254.01.patch |
| Optional Tests | site javadoc javac unit |
| git revision | trunk / cf23f2c |
| hadoop-common test log | https://builds.apache.org/job/PreCommit-HDFS-Build/13025/artifact/patchprocess/testrun_hadoop-common.txt |
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/13025/artifact/patchprocess/testrun_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/13025/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/13025/console |
This message was automatically generated.
Documentation-only patch, needs no new tests. Test failures are unrelated to the patch.
I'd thought "underdocumented" is a complete summary of kerberos info —good to see you trying to fix this.
in the patch, you say that `dn@REALM.TLD` is allowed. I recall seeing some JIRAs where people were saying you get a stack trace unless you have the /HOST value of some kind or other
The test failures do seem to be caused by my patch, oddly. I'll take a look.
in the patch, you say that `dn@REALM.TLD` is allowed. I recall seeing some JIRAs where people were saying you get a stack trace unless you have the /HOST value of some kind or other
Let me verify that in a test cluster, thanks for looking at the patch.
So yes it looks like at least the SaslRpcClient doesn't like principals without a host component.
192.168.56.80:8485: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Kerberos principal name does NOT have the expected hostname part: jn@EXAMPLE.COM; Host Details : local host is: "cm0.example.com/192.168.56.80"; destination host is: "cm0.example.com":8485; at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81) at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:232) at org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:899)
Whereas SecurityUtil handles them fine. We should be consistent. I'll file a separate bug to fix the SaslRpcClient, and any other components I run into, but also update the doc patch for now. Thanks for the catch.
v3 patch.
- I had fixed incorrect defaults in hdfs-defaults.xml which broke some secure mode tests. Fixed those tests to initialize their configuration correctly.
- Remove mention of ServiceName@REALM.TLD principals.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 6s | docker + precommit patch detected. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | mvninstall | 3m 13s | trunk passed |
| +1 | compile | 4m 23s | trunk passed with JDK v1.8.0_60 |
| +1 | compile | 4m 6s | trunk passed with JDK v1.7.0_79 |
| +1 | checkstyle | 0m 55s | trunk passed |
| +1 | mvnsite | 1m 34s | trunk passed |
| +1 | mvneclipse | 0m 28s | trunk passed |
| -1 | findbugs | 1m 51s | hadoop-hdfs-project/hadoop-hdfs in trunk cannot run convertXmlToText from findbugs |
| +1 | javadoc | 2m 0s | trunk passed with JDK v1.8.0_60 |
| +1 | javadoc | 2m 53s | trunk passed with JDK v1.7.0_79 |
| +1 | mvninstall | 2m 15s | the patch passed |
| +1 | compile | 4m 20s | the patch passed with JDK v1.8.0_60 |
| +1 | javac | 4m 20s | the patch passed |
| +1 | compile | 4m 17s | the patch passed with JDK v1.7.0_79 |
| +1 | javac | 4m 17s | the patch passed |
| +1 | checkstyle | 0m 59s | the patch passed |
| +1 | mvnsite | 1m 32s | the patch passed |
| +1 | mvneclipse | 0m 27s | the patch passed |
| -1 | whitespace | 0m 0s | The patch has 34 line(s) with tabs. |
| +1 | xml | 0m 0s | The patch has no ill-formed XML file. |
| +1 | findbugs | 3m 48s | the patch passed |
| +1 | javadoc | 1m 58s | the patch passed with JDK v1.8.0_60 |
| +1 | javadoc | 2m 48s | the patch passed with JDK v1.7.0_79 |
| -1 | unit | 6m 4s | hadoop-common in the patch failed with JDK v1.8.0_60. |
| -1 | unit | 62m 21s | hadoop-hdfs in the patch failed with JDK v1.8.0_60. |
| +1 | unit | 6m 49s | hadoop-common in the patch passed with JDK v1.7.0_79. |
| -1 | unit | 59m 4s | hadoop-hdfs in the patch failed with JDK v1.7.0_79. |
| -1 | asflicense | 0m 24s | Patch generated 58 ASF License warnings. |
| 181m 30s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_60 Failed junit tests | hadoop.security.TestShellBasedIdMapping |
| hadoop.hdfs.server.namenode.TestRecoverStripedBlocks | |
| hadoop.hdfs.server.datanode.TestDataNodeMetrics | |
| hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 | |
| hadoop.hdfs.TestDFSStripedOutputStream | |
| hadoop.hdfs.server.datanode.TestReadOnlySharedStorage | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | |
| hadoop.hdfs.TestReplication | |
| hadoop.hdfs.server.namenode.TestAddStripedBlocks | |
| hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes | |
| hadoop.hdfs.TestSafeModeWithStripedFile | |
| hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage | |
| hadoop.hdfs.server.namenode.TestAddBlockRetry | |
| hadoop.hdfs.server.namenode.TestAddBlock | |
| hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | |
| hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup | |
| hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 | |
| hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks | |
| hadoop.hdfs.server.mover.TestMover | |
| hadoop.hdfs.TestFileCreation | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 | |
| hadoop.hdfs.server.datanode.TestHSync | |
| hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA | |
| hadoop.hdfs.server.namenode.TestFsck | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 | |
| hadoop.hdfs.server.namenode.TestDecommissioningStatus | |
| hadoop.hdfs.server.blockmanagement.TestPendingReplication | |
| hadoop.fs.permission.TestStickyBit | |
| hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock | |
| hadoop.hdfs.server.balancer.TestBalancer | |
| hadoop.hdfs.server.blockmanagement.TestReplicationPolicy | |
| hadoop.hdfs.TestBlockStoragePolicy | |
| hadoop.hdfs.TestGetFileChecksum | |
| hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 | |
| hadoop.hdfs.TestDecommission | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 | |
| hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant | |
| hadoop.hdfs.server.datanode.TestDiskError | |
| hadoop.hdfs.TestReadStripedFileWithDecoding | |
| hadoop.hdfs.TestRecoverStripedFile | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | |
| hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN | |
| hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS | |
| hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork | |
| hadoop.hdfs.server.namenode.TestFSImageWithSnapshot | |
| hadoop.hdfs.server.blockmanagement.TestBlockManager | |
| hadoop.hdfs.TestReadWhileWriting | |
| hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain | |
| hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 | |
| hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks | |
| hadoop.hdfs.server.namenode.TestFileTruncate | |
| hadoop.hdfs.TestReplaceDatanodeOnFailure | |
| hadoop.hdfs.server.namenode.ha.TestHAAppend | |
| hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate | |
| hadoop.hdfs.TestCrcCorruption | |
| hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport | |
| hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | |
| hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | |
| hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer | |
| hadoop.hdfs.TestWriteReadStripedFile | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
| hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot | |
| hadoop.hdfs.TestQuota | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 | |
| hadoop.hdfs.TestInjectionForSimulatedStorage | |
| JDK v1.8.0_60 Timed out junit tests | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| JDK v1.7.0_79 Failed junit tests | hadoop.hdfs.server.namenode.TestRecoverStripedBlocks |
| hadoop.hdfs.server.datanode.TestDataNodeMetrics | |
| hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 | |
| hadoop.hdfs.TestDFSStripedOutputStream | |
| hadoop.hdfs.server.datanode.TestReadOnlySharedStorage | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | |
| hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages | |
| hadoop.hdfs.server.namenode.TestAddStripedBlocks | |
| hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes | |
| hadoop.hdfs.TestSafeModeWithStripedFile | |
| hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage | |
| hadoop.hdfs.server.namenode.TestAddBlockRetry | |
| hadoop.hdfs.server.namenode.ha.TestStandbyIsHot | |
| hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | |
| hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup | |
| hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 | |
| hadoop.hdfs.server.namenode.TestSnapshotPathINodes | |
| hadoop.hdfs.server.datanode.TestDeleteBlockPool | |
| hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks | |
| hadoop.hdfs.server.mover.TestMover | |
| hadoop.hdfs.TestFileCreation | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 | |
| hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 | |
| hadoop.hdfs.server.datanode.TestHSync | |
| hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA | |
| hadoop.hdfs.server.namenode.TestFsck | |
| hadoop.hdfs.server.blockmanagement.TestNodeCount | |
| hadoop.hdfs.server.namenode.TestDecommissioningStatus | |
| hadoop.hdfs.server.blockmanagement.TestPendingReplication | |
| hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock | |
| hadoop.hdfs.server.balancer.TestBalancer | |
| hadoop.hdfs.server.blockmanagement.TestReplicationPolicy | |
| hadoop.hdfs.TestBlockStoragePolicy | |
| hadoop.hdfs.TestGetFileChecksum | |
| hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 | |
| hadoop.hdfs.TestDecommission | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 | |
| hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant | |
| hadoop.hdfs.server.datanode.TestDiskError | |
| hadoop.hdfs.TestReadStripedFileWithDecoding | |
| hadoop.hdfs.TestRecoverStripedFile | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | |
| hadoop.hdfs.server.namenode.TestMetaSave | |
| hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS | |
| hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork | |
| hadoop.hdfs.server.namenode.TestFSImageWithSnapshot | |
| hadoop.hdfs.TestEncryptedTransfer | |
| hadoop.hdfs.server.blockmanagement.TestBlockManager | |
| hadoop.hdfs.server.namenode.ha.TestDNFencing | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 | |
| hadoop.hdfs.server.namenode.TestFileTruncate | |
| hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy | |
| hadoop.hdfs.TestReplaceDatanodeOnFailure | |
| hadoop.hdfs.server.namenode.ha.TestHAAppend | |
| hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate | |
| hadoop.hdfs.TestCrcCorruption | |
| hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 | |
| hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | |
| hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer | |
| hadoop.hdfs.TestWriteReadStripedFile | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 | |
| hadoop.hdfs.TestInjectionForSimulatedStorage | |
| JDK v1.7.0_79 Timed out junit tests | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
This message was automatically generated.
- check the indentation on some of the XML examples
- doesn't my little ebook merit a reference?
- you can paste in it's references too, or at least a subset of them: https://github.com/steveloughran/kerberos_and_hadoop/blob/master/sections/biblography.md
Thanks for the review. I've added a references section including your book and fixed the tabs issue in hdfs-default.xml.
I referred to your book when I was stuck debugging SPNEGO auth issues recently but didn't find an answer that helped me. Finally tracked it down to DNS-related quirks. It prompted me to document the parts I understood but I still don't understand SPNEGO well enough to improve its documentation. I don't envy the next person who has to debug Hadoop security quirks.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 9s | docker + precommit patch detected. |
| +1 | @author | 0m 1s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | mvninstall | 3m 15s | trunk passed |
| +1 | compile | 5m 19s | trunk passed with JDK v1.8.0_66 |
| +1 | compile | 5m 33s | trunk passed with JDK v1.7.0_79 |
| +1 | checkstyle | 1m 3s | trunk passed |
| +1 | mvnsite | 1m 58s | trunk passed |
| +1 | mvneclipse | 0m 46s | trunk passed |
| +1 | findbugs | 5m 4s | trunk passed |
| +1 | javadoc | 2m 52s | trunk passed with JDK v1.8.0_66 |
| +1 | javadoc | 3m 20s | trunk passed with JDK v1.7.0_79 |
| +1 | mvninstall | 2m 23s | the patch passed |
| +1 | compile | 5m 14s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 5m 14s | the patch passed |
| +1 | compile | 4m 55s | the patch passed with JDK v1.7.0_79 |
| +1 | javac | 4m 55s | the patch passed |
| +1 | checkstyle | 1m 6s | the patch passed |
| +1 | mvnsite | 1m 51s | the patch passed |
| +1 | mvneclipse | 0m 32s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 | findbugs | 4m 35s | the patch passed |
| +1 | javadoc | 2m 30s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 3m 19s | the patch passed with JDK v1.7.0_79 |
| -1 | unit | 8m 44s | hadoop-common in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 78m 53s | hadoop-hdfs in the patch failed with JDK v1.8.0_66. |
| +1 | unit | 9m 13s | hadoop-common in the patch passed with JDK v1.7.0_79. |
| -1 | unit | 71m 3s | hadoop-hdfs in the patch failed with JDK v1.7.0_79. |
| -1 | asflicense | 0m 22s | Patch generated 56 ASF License warnings. |
| 225m 29s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_66 Failed junit tests | hadoop.metrics2.impl.TestMetricsSystemImpl |
| hadoop.ipc.TestRPCWaitForProxy | |
| hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
| hadoop.hdfs.server.namenode.ha.TestEditLogTailer | |
| hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | |
| hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | |
| hadoop.hdfs.TestRollingUpgrade | |
| hadoop.hdfs.server.datanode.TestDirectoryScanner | |
| JDK v1.7.0_79 Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| hadoop.hdfs.shortcircuit.TestShortCircuitCache | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | |
| hadoop.hdfs.TestDFSClientRetries |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 6s | docker + precommit patch detected. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
| +1 | mvninstall | 2m 59s | trunk passed |
| +1 | compile | 4m 23s | trunk passed with JDK v1.8.0_60 |
| +1 | compile | 4m 15s | trunk passed with JDK v1.7.0_79 |
| +1 | checkstyle | 0m 57s | trunk passed |
| +1 | mvnsite | 1m 36s | trunk passed |
| +1 | mvneclipse | 0m 29s | trunk passed |
| +1 | findbugs | 3m 33s | trunk passed |
| +1 | javadoc | 2m 2s | trunk passed with JDK v1.8.0_60 |
| +1 | javadoc | 2m 50s | trunk passed with JDK v1.7.0_79 |
| +1 | mvninstall | 2m 12s | the patch passed |
| +1 | compile | 4m 26s | the patch passed with JDK v1.8.0_60 |
| +1 | javac | 4m 26s | the patch passed |
| +1 | compile | 4m 12s | the patch passed with JDK v1.7.0_79 |
| +1 | javac | 4m 12s | the patch passed |
| +1 | checkstyle | 0m 56s | the patch passed |
| +1 | mvnsite | 1m 36s | the patch passed |
| +1 | mvneclipse | 0m 28s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 | findbugs | 3m 49s | the patch passed |
| +1 | javadoc | 2m 1s | the patch passed with JDK v1.8.0_60 |
| +1 | javadoc | 2m 54s | the patch passed with JDK v1.7.0_79 |
| -1 | unit | 16m 58s | hadoop-common in the patch failed with JDK v1.8.0_60. |
| -1 | unit | 51m 47s | hadoop-hdfs in the patch failed with JDK v1.8.0_60. |
| -1 | unit | 7m 40s | hadoop-common in the patch failed with JDK v1.7.0_79. |
| -1 | unit | 52m 53s | hadoop-hdfs in the patch failed with JDK v1.7.0_79. |
| -1 | asflicense | 0m 20s | Patch generated 56 ASF License warnings. |
| 176m 39s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_60 Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
| hadoop.hdfs.server.namenode.ha.TestDNFencing | |
| hadoop.hdfs.server.blockmanagement.TestNodeCount | |
| hadoop.hdfs.server.namenode.TestDecommissioningStatus | |
| hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes | |
| JDK v1.8.0_60 Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
| JDK v1.7.0_79 Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
| hadoop.fs.TestLocalFsFCStatistics | |
| hadoop.hdfs.server.blockmanagement.TestNodeCount | |
| hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
This message was automatically generated.
The failures look unrelated to the patch.
Steve Loughran, any way I can convince you to review the latest patch? I think it addresses your feedback.
yeah, find a way to make my ZK client code not to leave GSS API error 44 in the zk logs
I have no idea what that's about. So I am going to see if I can find another reviewer.
Some of the default values in tables are emphasised; others converted to `code` font. I think they should all be `in code`.
You can do `italiccode` for both
line 271,
/etc/security/keytab/nn.service.keytab should be ` /etc/security/keytab/nn.service.keytab`
same for line 273 and /etc/security/keytab/spnego.service.keytab
... etc
otherwise, +1
Nice improvements. Thank you, Arpit.
In the references, the first entry can link to HADOOP-4487, which is the original Hadoop security JIRA with design doc attached.
In HdfsMultiHoming.md, the "Hostname Lookup" link isn't rendering correctly. There is a missing underscore. This would fix it:
* [Hostname Lookup](#Hostname_Lookup)
The descriptions in hdfs-default.xml are nice to have, but I'm concerned that they are not backwards-compatible. dfs.namenode.kerberos.internal.spnego.principal used to default to using the value of dfs.web.authentication.kerberos.principal. That would mean that if someone had configured dfs.web.authentication.kerberos.principal, but not dfs.namenode.kerberos.internal.spnego.principal, then it could have worked before this patch, but it would stop working after this patch.
That probably partly explains the earlier SaslDataTransferTestCase failures. I think if tests need to be changed for this patch, then it's probably a sign that something has gone wrong.
Hi Chris Nauroth, thanks for the review!
The descriptions in hdfs-default.xml are nice to have, but I'm concerned that they are not backwards-compatible.
There should be no incompatibility introduced by the patch. It just fixes wrong documentation to be in line with the code. e.g. dfs.namenode.kerberos.internal.spnego.principal has no default in code but it was documented as defaulting to dfs.web.authentication.kerberos.principal in hdfs-default.xml. From NameNodeHttpServer#start:
HttpServer2.Builder builder = DFSUtil.httpServerTemplateForNNAndJN(conf,
httpAddr, httpsAddr, "hdfs",
DFSConfigKeys.DFS_NAMENODE_KERBEROS_INTERNAL_SPNEGO_PRINCIPAL_KEY,
DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY);
A few unit tests broke after this patch because they were pulling hdfs-default.xml as a resource and depending on the incorrect defaults instead of initializing their own test configuration correctly.
I'll post an updated patch addressing the remaining feedback.
Taking this example from the existing hdfs-default.xml:
<property>
<name>dfs.namenode.kerberos.internal.spnego.principal</name>
<value>${dfs.web.authentication.kerberos.principal}</value>
</property>
If an hdfs-site.xml defined dfs.web.authentication.kerberos.principal, but did not define dfs.namenode.kerberos.internal.spnego.principal, then I think it still would have worked because of the token substitution logic in Configuration#substituteVars. In this patch, dfs.namenode.kerberos.internal.spnego.principal defaults to nothing, so the token substitution won't happen anymore. This is why I think there could be a compatibility problem.
If an hdfs-site.xml defined dfs.web.authentication.kerberos.principal, but did not define dfs.namenode.kerberos.internal.spnego.principal, then I think it still would have worked because of the token substitution logic in Configuration#substituteVars.
Nice catch Chris! So the reason this works is because hdfs-default.xml is available as a resource in the hadoop-hdfs jar.
I'll update the next patch to restore the defaults.
v05 patch addresses feedback from Steve:
- Made formatting of the default values consistent. A lot of the existing settings were inconsistently formatted so fixed those too, hence the increase in patch size.
And Chris:
- Restored defaults to hdfs-default.xml and reverted test changes
- Fixed broken hyperlink in HdfsMultihoming.md, added a link to the
HADOOP-4487design doc.
Also pretty-printed the markdown tables as they were getting messy to read.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 0s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 7m 54s | trunk passed |
| +1 | compile | 8m 5s | trunk passed with JDK v1.8.0_66 |
| +1 | compile | 8m 40s | trunk passed with JDK v1.7.0_91 |
| +1 | mvnsite | 1m 55s | trunk passed |
| +1 | mvneclipse | 0m 28s | trunk passed |
| +1 | javadoc | 2m 2s | trunk passed with JDK v1.8.0_66 |
| +1 | javadoc | 2m 48s | trunk passed with JDK v1.7.0_91 |
| +1 | mvninstall | 2m 33s | the patch passed |
| +1 | compile | 7m 59s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 7m 59s | the patch passed |
| +1 | compile | 8m 43s | the patch passed with JDK v1.7.0_91 |
| +1 | javac | 8m 43s | the patch passed |
| +1 | mvnsite | 1m 53s | the patch passed |
| +1 | mvneclipse | 0m 28s | the patch passed |
| -1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 | javadoc | 2m 1s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 2m 51s | the patch passed with JDK v1.7.0_91 |
| +1 | unit | 7m 25s | hadoop-common in the patch passed with JDK v1.8.0_66. |
| -1 | unit | 51m 16s | hadoop-hdfs in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 7m 6s | hadoop-common in the patch failed with JDK v1.7.0_91. |
| -1 | unit | 49m 59s | hadoop-hdfs in the patch failed with JDK v1.7.0_91. |
| +1 | asflicense | 0m 27s | Patch does not generate ASF License warnings. |
| 175m 30s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| hadoop.hdfs.server.namenode.TestDecommissioningStatus | |
| hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes | |
| hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits | |
| JDK v1.7.0_91 Failed junit tests | hadoop.http.TestHttpServer |
| hadoop.hdfs.TestRenameWhileOpen |
This message was automatically generated.
+1 for patch v05. I have committed this to trunk, branch-2 and branch-2.8. Arpit, thank you for the documentation improvements. Steve, thank you for reviewing.
Thank you Steve and Chris.
FAILURE: Integrated in Hadoop-trunk-Commit #9002 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9002/)
HADOOP-12570. HDFS Secure Mode Documentation updates. Contributed by (cnauroth: rev 932956458dddabd9583b911c693386a56d723c0d)
- hadoop-common-project/hadoop-common/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsMultihoming.md
- hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md
- hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
- hadoop-common-project/hadoop-common/src/site/markdown/HttpAuthentication.md
Cherry-picked this to 2.7.3.
Closing the JIRA as part of 2.7.3 release.
v1 patch.
HADOOP-12437.