Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3813

Log error message if security and WebHDFS are enabled but principal/keytab are not configured

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.3-alpha
    • Component/s: security, webhdfs
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      I configured a secure HDFS cluster, but failed to start the NameNode because I had enabled WebHDFS without specifying dfs.web.authentication.kerberos.principal in hdfs-site.xml.

      In the NN logs, I saw:

      2012-05-28 17:50:13,021 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Login using keytab /etc/hdfs.keytab, for principal HTTP/c1225.hal.cloudera.com@HAL.CLOUDERA.COM
      2012-05-28 17:50:13,030 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: Initialized, principal [HTTP/c1225.hal.cloudera.com@HAL.CLOUDERA.COM] from keytab [/etc/hdfs.keytab]
      2012-05-28 17:50:13,031 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: 'signature.secret' configuration not set, using a random value as secret
      2012-05-28 17:50:13,032 WARN org.mortbay.log: failed SPNEGO: javax.servlet.ServletException: javax.servlet.ServletException: Principal not defined in configuration
      2012-05-28 17:50:13,033 WARN org.mortbay.log: Failed startup of context org.mortbay.jetty.webapp.WebAppContext@21453d72{/,file:/usr/lib/hadoop-hdfs/webapps/hdfs}
      javax.servlet.ServletException: javax.servlet.ServletException: Principal not defined in configuration
      	at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:185)
      	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:146)
      	at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
      	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
      	at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
      	at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
      	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
      	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
      	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
      	at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
      	at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
      	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
      	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      	at org.mortbay.jetty.Server.doStart(Server.java:224)
      	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
      	at org.apache.hadoop.http.HttpServer.start(HttpServer.java:617)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:173)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:529)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:471)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:434)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:590)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:571)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1134)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1193)
      Caused by: javax.servlet.ServletException: Principal not defined in configuration
      	at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:146)
      	... 24 more
      2012-05-28 17:50:13,034 WARN org.mortbay.log: Nested in javax.servlet.ServletException: javax.servlet.ServletException: Principal not defined in configuration:
      javax.servlet.ServletException: Principal not defined in configuration
      	at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:146)
      	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:146)
      	at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
      	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
      	at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
      	at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
      	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
      	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
      	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
      	at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
      	at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
      	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
      	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      	at org.mortbay.jetty.Server.doStart(Server.java:224)
      	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
      	at org.apache.hadoop.http.HttpServer.start(HttpServer.java:617)
      	at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:173)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:529)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:471)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:434)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:590)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:571)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1134)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1193)
      2012-05-28 17:50:13,041 INFO org.mortbay.log: Started SelectChannelConnector@c1225.hal.cloudera.com:50070
      2012-05-28 17:50:13,041 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: c1225.hal.cloudera.com:50070
      2012-05-28 17:50:13,042 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
      2012-05-28 17:50:13,042 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 17020: starting
      2012-05-28 17:50:13,045 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at: c1225.hal.cloudera.com/172.29.98.216:17020
      2012-05-28 17:50:13,045 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for standby state
      2012-05-28 17:50:13,048 INFO org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Will roll logs on active node at c1226.hal.cloudera.com/172.29.98.217:17020 every 120 seconds.
      2012-05-28 17:50:13,058 INFO org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer: Starting standby checkpoint thread...
      Checkpointing active NN at c1226.hal.cloudera.com:50070
      Serving checkpoints at c1225.hal.cloudera.com/172.29.98.216:50070
      

      I couldn't figure out what I had misconfigured, but ATM found that I was missing dfs.web.authentication.kerberos.principal.

      Logging an error if this property is not configured when WebHDFS and security are enabled would be useful for future users running into the same problem.

        Attachments

        1. error_output
          30 kB
          Stephen Chu
        2. HDFS-3813.patch
          2 kB
          Stephen Chu
        3. HDFS-3813.patch
          1 kB
          Stephen Chu

        Issue Links

          Activity

            People

            • Assignee:
              schu Stephen Chu
              Reporter:
              schu Stephen Chu

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment