Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 0.20.205.0
-
Fix Version/s: 0.20.205.0, 0.23.0, 0.24.0
-
Component/s: None
-
Labels:None
-
Hadoop Flags:Reviewed
Description
the following defaults are set in hdfs-defaults.xml
<property>
<name>dfs.web.authentication.kerberos.principal</name>
<value>HTTP/$
@$
{kerberos.realm}</value>
<description>
The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.
The HTTP Kerberos principal MUST start with 'HTTP/' per Kerberos
HTTP SPENGO specification.
</description>
</property>
<property>
<name>dfs.web.authentication.kerberos.keytab</name>
<value>$
/dfs.web.keytab</value>
<description>
The Kerberos keytab file with the credentials for the
HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.
</description>
</property>
These properties should not have defaults
Issue Links
- is related to
-
HDFS-2316
[umbrella] WebHDFS: a complete FileSystem implementation for accessing HDFS over HTTP
-
- Resolved
-
-
HADOOP-7688
When a servlet filter throws an exception in init(..), the Jetty server failed silently.
-
- Closed
-
Activity
For the security conf properties, we should do the same as other HDFS security properties.
Jetty caught the exception and simply printed in the log. I will see if we can change this.
Just found that we have similar conf properties in core-default.xml for Kerberos ssl.
<property>
<name>hadoop.http.authentication.kerberos.principal</name>
<value>HTTP/localhost@LOCALHOST</value>
<description>
Indicates the Kerberos principal to be used for HTTP endpoint.
The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO specification.
</description>
</property>
<property>
<name>hadoop.http.authentication.kerberos.keytab</name>
<value>${user.home}/hadoop.keytab</value>
<description>
Location of the keytab file with the credentials for the principal.
Referring to the same keytab file Oozie uses for its Kerberos credentials for Hadoop.
</description>
</property>
Nicholas, so is this not-a-bug?
Yes, this is not a bug but an improvement.
also the namenode ui does not come up if the configs for the kerberos do not have the correct values. So we should put this in release notes until this is fixed.
The other security related conf properties (e.g. dfs.namenode.kerberos.principal) are found in hdfs-site.xml. Let's do the same for SPNEGO.
h2368_20110927_0.20s.patch
h2368_20110927.patch
Move the SPNEGO conf properties from hdfs-default.xml to hdfs-site.xml
+1 for the patch.
I think that namenode should use combined keytab for http principal as well. Currently, namenode uses same key tab for both namenode principal and https(host) principal. But, that would require some code change as well and can be addressed in a separate jira.
Thanks Jitendra for the review.
I have committed this.
Integrated in Hadoop-Hdfs-trunk-Commit #1050 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1050/)
HDFS-2368. Move SPNEGO conf properties from hdfs-default.xml to hdfs-site.xml.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176719
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Integrated in Hadoop-Common-trunk-Commit #972 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/972/)
HDFS-2368. Move SPNEGO conf properties from hdfs-default.xml to hdfs-site.xml.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176719
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Integrated in Hadoop-Mapreduce-trunk-Commit #993 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/993/)
HDFS-2368. Move SPNEGO conf properties from hdfs-default.xml to hdfs-site.xml.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176719
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
... Also should the NN start should fail if this happens. Currently when the nn principal login fails the namenode fails to start. We should implement the same for this.
This is a general problem in our HttpServer; filed HADOOP-7688.
Integrated in Hadoop-Hdfs-trunk #814 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/814/)
HDFS-2368. Move SPNEGO conf properties from hdfs-default.xml to hdfs-site.xml.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176719
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Integrated in Hadoop-Mapreduce-trunk #844 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/844/)
HDFS-2368. Move SPNEGO conf properties from hdfs-default.xml to hdfs-site.xml.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176719
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hdfs-site.xml
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Closed upon release of 0.20.205.0
Merged to 0.23.
Integrated in Hadoop-Common-0.23-Commit #61 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/61/)
svn merge -c 1176719 from trunk for HDFS-2368.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189435
Files :
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Integrated in Hadoop-Hdfs-0.23-Commit #62 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/62/)
svn merge -c 1176719 from trunk for HDFS-2368.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189435
Files :
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Integrated in Hadoop-Mapreduce-0.23-Commit #58 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/58/)
svn merge -c 1176719 from trunk for HDFS-2368.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189435
Files :
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Integrated in Hadoop-Hdfs-0.23-Build #52 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/52/)
svn merge -c 1176719 from trunk for HDFS-2368.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189435
Files :
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Integrated in Hadoop-Mapreduce-0.23-Build #64 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/64/)
svn merge -c 1176719 from trunk for HDFS-2368.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189435
Files :
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
- /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
currently because of these defaults the namenode tries to log in and fails
2011-09-26 21:29:14,532 WARN org.mortbay.log: Failed startup of context org.mortbay.jetty.webapp.WebAppContext@1b5a5cf
{/,file:/hadoop-0.20.205.0/webapps/hdfs}javax.servlet.ServletException: javax.servlet.ServletException: Keytab does not exist: /homes/user/dfs.web.keytab
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:180)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:146)
at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:617)
at org.apache.hadoop.hdfs.server.namenode.NameNode$1.run(NameNode.java:421)
at org.apache.hadoop.hdfs.server.namenode.NameNode$1.run(NameNode.java:351)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:351)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:472)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1243)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1252)
Caused by: javax.servlet.ServletException: Keytab does not exist: /homes/hrt_hdfs/dfs.web.keytab
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:151)
We should not have these properties as defaults but expect the user to explicitly set them when they enable webhdfs. Also should the NN start should fail if this happens. Currently when the nn principal login fails the namenode fails to start. We should implement the same for this.