Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.4.6
    • Fix Version/s: None
    • Component/s: server
    • Labels:
      None

      Description

      When there's a problem authenticating in {{ServerCnxnFactory.configureSaslLogin() }}, the exception is retained -but the full stack lost.

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -
          Show
          stevel@apache.org Steve Loughran added a comment - + linked to it from the ZK section of the hadoop & kerberos ebook
          Hide
          stevel@apache.org Steve Loughran added a comment -

          good summary of tactics.

          If you have a look at HADOOP-12426, we've a new class, KDiag, to go into Hadoop core to do a big chunk of this diagnostics. Though not looking in the JAAS file to extract its principal -nice idea. Network diagnostics is something we should do too, probably initially independently.

          The KDiag patch has yet to go in/get much public review —if you do hit kerberos problems, then try and let us know how you got on. And any extra diagnostics you can add will be most welcome.

          Show
          stevel@apache.org Steve Loughran added a comment - good summary of tactics. If you have a look at HADOOP-12426 , we've a new class, KDiag, to go into Hadoop core to do a big chunk of this diagnostics. Though not looking in the JAAS file to extract its principal -nice idea. Network diagnostics is something we should do too, probably initially independently. The KDiag patch has yet to go in/get much public review —if you do hit kerberos problems, then try and let us know how you got on. And any extra diagnostics you can add will be most welcome.
          Hide
          jcustenborder Jeremy Custenborder added a comment -

          I've bumped into this a couple times recently so I'll add some troubleshooting information for anyone who arrives here via a search for the exception.

          Verify DNS

          The machine must be able to resolve it's own hostname to the hostname that other machines will connect to. Verify the hosts file is setup correctly.

          Verify the format of your jaas file

          The principal needs to be to the fully qualified hostname

          Server {
            com.sun.security.auth.module.Krb5LoginModule required
            useKeyTab=true
            keyTab="/etc/security/keytabs/zookeeper.keytab"
            storeKey=true
            useTicketCache=false
            principal="zookeeper/zookeeper.example.com@EXAMPLE.COM";
          };
          

          Make sure you can kinit with the contents of your jaas file

          If possible you should try to do this with the local user zookeeper will be running as.

          kinit -kt <keyTab value from jaas> <zookeeper/zookeeper.example.com@EXAMPLE.COM>
          

          This should return successfully without prompting for a password.

          Make sure your jaas file and keytab are readable by the zookeeper user

          ls -la 
          -r--r-----. 1 zookeeper zookeeper 4198 Feb  3 19:32 /etc/security/keytabs/zookeeper.keytab
          

          If the permissions are not correct use this.

          chmod 0440 /etc/security/keytabs/zookeeper.keytab
          

          Verify you have the proper JCE policy files (Oracle JRE)

          If you are using the Oracle JRE and strong encryption like aes256-cts-hmac-sha1-96 or aes128-cts-hmac-sha1-96, additional policy jars need to be installed. Download the JCE jars for Java 7 or Java 8

          Show
          jcustenborder Jeremy Custenborder added a comment - I've bumped into this a couple times recently so I'll add some troubleshooting information for anyone who arrives here via a search for the exception. Verify DNS The machine must be able to resolve it's own hostname to the hostname that other machines will connect to. Verify the hosts file is setup correctly. Verify the format of your jaas file The principal needs to be to the fully qualified hostname Server { com.sun.security.auth.module.Krb5LoginModule required useKeyTab= true keyTab= "/etc/security/keytabs/zookeeper.keytab" storeKey= true useTicketCache= false principal= "zookeeper/zookeeper.example.com@EXAMPLE.COM" ; }; Make sure you can kinit with the contents of your jaas file If possible you should try to do this with the local user zookeeper will be running as. kinit -kt <keyTab value from jaas> <zookeeper/zookeeper.example.com@EXAMPLE.COM> This should return successfully without prompting for a password. Make sure your jaas file and keytab are readable by the zookeeper user ls -la -r--r-----. 1 zookeeper zookeeper 4198 Feb 3 19:32 /etc/security/keytabs/zookeeper.keytab If the permissions are not correct use this. chmod 0440 /etc/security/keytabs/zookeeper.keytab Verify you have the proper JCE policy files (Oracle JRE) If you are using the Oracle JRE and strong encryption like aes256-cts-hmac-sha1-96 or aes128-cts-hmac-sha1-96, additional policy jars need to be installed. Download the JCE jars for Java 7 or Java 8
          Hide
          stevel@apache.org Steve Loughran added a comment -

          That first warning is a sign that username == null in the sasl callback, which is a more serious condition: a password callback has come in without a name callback, or the name callback came in but {{credentials.get(nc.getDefaultName()} == null} . Either way, it could be reported better.

          Show
          stevel@apache.org Steve Loughran added a comment - That first warning is a sign that username == null in the sasl callback, which is a more serious condition: a password callback has come in without a name callback, or the name callback came in but {{credentials.get(nc.getDefaultName()} == null} . Either way, it could be reported better.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          2015-12-15 17:16:23,517 - WARN  [main:SaslServerCallbackHandler@105] - No password found for user: null
          2015-12-15 17:16:23,536 - ERROR [main:ZooKeeperServerMain@63] - Unexpected exception, exiting abnormallyjava.io.IOException: Could not configure server because SASL configuration did not allow the  ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: No password provided
                  at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207)
                  at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)
                  at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:111)
                  at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
                  at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
                  at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
                  at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
          

          Now, there's clearly a hint of the root cause "No password found for user: null", but the inner stack is lost, and more info could be provided

          Show
          stevel@apache.org Steve Loughran added a comment - 2015-12-15 17:16:23,517 - WARN [main:SaslServerCallbackHandler@105] - No password found for user: null 2015-12-15 17:16:23,536 - ERROR [main:ZooKeeperServerMain@63] - Unexpected exception, exiting abnormallyjava.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: No password provided at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:111) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) Now, there's clearly a hint of the root cause "No password found for user: null", but the inner stack is lost, and more info could be provided

            People

            • Assignee:
              Unassigned
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development