Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-2040

Server to log underlying cause of SASL connection problems

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.4.6
    • 3.4.7, 3.5.2, 3.6.0
    • server
    • None
    • Reviewed

    Description

      When you have SASL connectivity problems, you spend time staring at logs —ideally logs with stack traces.

      ZK server can help here by including the stack traces when there is a SASL auth problem, rather than just giving the text of the exception.

      Attachments

        1. ZOOKEEPER-2040.002.patch
          0.8 kB
          Chris Nauroth
        2. ZOOKEEPER-2040-log-SASL-errors-001.patch
          0.8 kB
          Steve Loughran

        Activity

          stevel@apache.org Steve Loughran added a comment -

          adds the underlying exception.

          Before

          WARN  server.ZooKeeperServer (ZooKeeperServer.java:processSasl(969)) - Client failed to SASL authenticate: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
          

          After

           WARN  server.ZooKeeperServer (ZooKeeperServer.java:processSasl(969)) - Client failed to SASL authenticate: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
          javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]
          	at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177)
          	at org.apache.zookeeper.server.ZooKeeperSaslServer.evaluateResponse(ZooKeeperSaslServer.java:158)
          	at org.apache.zookeeper.server.ZooKeeperServer.processSasl(ZooKeeperServer.java:961)
          	at org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:934)
          	at org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373)
          	at org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200)
          	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244)
          	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
          	at java.lang.Thread.run(Thread.java:745)
          Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
          	at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788)
          	at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
          	at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
          	at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155)
          	... 8 more
          Caused by: KrbException: Checksum failed
          	at sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType.decrypt(Aes128CtsHmacSha1EType.java:102)
          	at sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType.decrypt(Aes128CtsHmacSha1EType.java:94)
          	at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:177)
          	at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:278)
          	at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:144)
          	at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
          	at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771)
          	... 11 more
          Caused by: java.security.GeneralSecurityException: Checksum failed
          	at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451)
          	at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272)
          	at sun.security.krb5.internal.crypto.Aes128.decrypt(Aes128.java:76)
          	at sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType.decrypt(Aes128CtsHmacSha1EType.java:100)
          	... 17 more
          

          It may seem noisier, but it's the information needed to actually work out what the problem is, here something AES related

          stevel@apache.org Steve Loughran added a comment - adds the underlying exception. Before WARN server.ZooKeeperServer (ZooKeeperServer.java:processSasl(969)) - Client failed to SASL authenticate: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)] After WARN server.ZooKeeperServer (ZooKeeperServer.java:processSasl(969)) - Client failed to SASL authenticate: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)] javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:177) at org.apache.zookeeper.server.ZooKeeperSaslServer.evaluateResponse(ZooKeeperSaslServer.java:158) at org.apache.zookeeper.server.ZooKeeperServer.processSasl(ZooKeeperServer.java:961) at org.apache.zookeeper.server.ZooKeeperServer.processPacket(ZooKeeperServer.java:934) at org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:373) at org.apache.zookeeper.server.NIOServerCnxn.readPayload(NIOServerCnxn.java:200) at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:244) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang. Thread .run( Thread .java:745) Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed) at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:788) at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:155) ... 8 more Caused by: KrbException: Checksum failed at sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType.decrypt(Aes128CtsHmacSha1EType.java:102) at sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType.decrypt(Aes128CtsHmacSha1EType.java:94) at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:177) at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:278) at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:144) at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108) at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:771) ... 11 more Caused by: java.security.GeneralSecurityException: Checksum failed at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451) at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272) at sun.security.krb5.internal.crypto.Aes128.decrypt(Aes128.java:76) at sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType.decrypt(Aes128CtsHmacSha1EType.java:100) ... 17 more It may seem noisier, but it's the information needed to actually work out what the problem is, here something AES related
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12669084/ZOOKEEPER-2040-log-SASL-errors-001.patch
          against trunk revision 1623916.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2338//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12669084/ZOOKEEPER-2040-log-SASL-errors-001.patch against trunk revision 1623916. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2338//console This message is automatically generated.
          cnauroth Chris Nauroth added a comment -

          +1 (non-binding) for this change. Thanks, Steve.

          cnauroth Chris Nauroth added a comment - +1 (non-binding) for this change. Thanks, Steve.
          cnauroth Chris Nauroth added a comment -

          rgs, would you please consider including Steve's patch in the upcoming 3.4.7 release? It's a one-line change to a log message, and it can be very useful while troubleshooting.

          I'm uploading a patch v002 file. This just takes Steve's earlier patch and regenerates it with the --no-prefix option to satisfy the ZooKeeper test-patch.sh script. This patch file can apply to both trunk and branch-3.4.

          cnauroth Chris Nauroth added a comment - rgs , would you please consider including Steve's patch in the upcoming 3.4.7 release? It's a one-line change to a log message, and it can be very useful while troubleshooting. I'm uploading a patch v002 file. This just takes Steve's earlier patch and regenerates it with the --no-prefix option to satisfy the ZooKeeper test-patch.sh script. This patch file can apply to both trunk and branch-3.4.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12730241/ZOOKEEPER-2040.002.patch
          against trunk revision 1677529.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2676//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2676//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2676//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12730241/ZOOKEEPER-2040.002.patch against trunk revision 1677529. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2676//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2676//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2676//console This message is automatically generated.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12730241/ZOOKEEPER-2040.002.patch
          against trunk revision 1677529.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2677//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2677//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2677//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12730241/ZOOKEEPER-2040.002.patch against trunk revision 1677529. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2677//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2677//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2677//console This message is automatically generated.
          cnauroth Chris Nauroth added a comment -

          I am still +1 for this patch, but now my +1 is binding. I plan to commit this to trunk, branch-3.5 and branch-3.4 on Thursday, 9/10, assuming there are no objections before then.

          cnauroth Chris Nauroth added a comment - I am still +1 for this patch, but now my +1 is binding. I plan to commit this to trunk, branch-3.5 and branch-3.4 on Thursday, 9/10, assuming there are no objections before then.

          (and sorry I missed this before).

          rgs Raúl Gutiérrez Segalés added a comment - (and sorry I missed this before).
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12730241/ZOOKEEPER-2040.002.patch
          against trunk revision 1702163.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2868//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2868//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2868//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12730241/ZOOKEEPER-2040.002.patch against trunk revision 1702163. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2868//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2868//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/2868//console This message is automatically generated.
          cnauroth Chris Nauroth added a comment -

          I have committed this to trunk, branch-3.5 and branch-3.4. stevel@apache.org, thank you for contributing the patch.

          cnauroth Chris Nauroth added a comment - I have committed this to trunk, branch-3.5 and branch-3.4. stevel@apache.org , thank you for contributing the patch.
          hudson Hudson added a comment -

          SUCCESS: Integrated in ZooKeeper-trunk #2798 (See https://builds.apache.org/job/ZooKeeper-trunk/2798/)
          ZOOKEEPER-2040: Server to log underlying cause of SASL connection problems. (Steve Loughran via cnauroth) (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1702378)

          • /zookeeper/trunk/CHANGES.txt
          • /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java
          hudson Hudson added a comment - SUCCESS: Integrated in ZooKeeper-trunk #2798 (See https://builds.apache.org/job/ZooKeeper-trunk/2798/ ) ZOOKEEPER-2040 : Server to log underlying cause of SASL connection problems. (Steve Loughran via cnauroth) (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1702378 ) /zookeeper/trunk/CHANGES.txt /zookeeper/trunk/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java

          People

            stevel@apache.org Steve Loughran
            stevel@apache.org Steve Loughran
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: