ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-1181

Fix problems with Kerberos TGT renewal

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      -Fixes two findbugs warnings related to holding a lock while sleeping.
      -Addresses Camille's point: merge two almost-identical retry methods into a single retry method.
      Show
      -Fixes two findbugs warnings related to holding a lock while sleeping. -Addresses Camille's point: merge two almost-identical retry methods into a single retry method.

      Description

      Currently, in Zookeeper trunk, there are two problems with Kerberos TGT renewal:

      1. TGTs obtained from a keytab are not refreshed periodically. They should be, just as those from ticket cache are refreshed.

      2. Ticket renewal should be retried if it fails. Ticket renewal might fail if two or more separate processes (different JVMs) running as the same user try to renew Kerberos credentials at the same time.

      1. ZOOKEEPER-1181.patch
        25 kB
        Eugene Koontz
      2. ZOOKEEPER-1181.patch
        25 kB
        Eugene Koontz
      3. ZOOKEEPER-1181.patch
        25 kB
        Mahadev konar

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in ZooKeeper-trunk #1342 (See https://builds.apache.org/job/ZooKeeper-trunk/1342/)
          ZOOKEEPER-1181. Fix problems with Kerberos TGT renewal. (Eugene Koontz via mahadev)

          mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188033
          Files :

          • /zookeeper/trunk/CHANGES.txt
          • /zookeeper/trunk/src/java/main/org/apache/zookeeper/Login.java
          Show
          Hudson added a comment - Integrated in ZooKeeper-trunk #1342 (See https://builds.apache.org/job/ZooKeeper-trunk/1342/ ) ZOOKEEPER-1181 . Fix problems with Kerberos TGT renewal. (Eugene Koontz via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188033 Files : /zookeeper/trunk/CHANGES.txt /zookeeper/trunk/src/java/main/org/apache/zookeeper/Login.java
          Hide
          Mahadev konar added a comment -

          Just committed this. Hope to see test cases soon for these classes. Thanks Eugene!

          Show
          Mahadev konar added a comment - Just committed this. Hope to see test cases soon for these classes. Thanks Eugene!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12499644/ZOOKEEPER-1181.patch
          against trunk revision 1185994.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/625//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/625//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/625//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499644/ZOOKEEPER-1181.patch against trunk revision 1185994. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/625//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/625//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/625//console This message is automatically generated.
          Hide
          Mahadev konar added a comment -

          Reuploading the same patch to run thorugh hudson.

          Show
          Mahadev konar added a comment - Reuploading the same patch to run thorugh hudson.
          Hide
          Eugene Koontz added a comment -

          Hi Mahadev, good idea; I added ZOOKEEPER-1205 for unit tests for this bug.

          Show
          Eugene Koontz added a comment - Hi Mahadev, good idea; I added ZOOKEEPER-1205 for unit tests for this bug.
          Hide
          Mahadev konar added a comment -

          Eugene,
          We should write some unit tests for this. I am fine checking this into 3.4 for now. Can you please create a ticket to add a unit test for this? Mockito would be very helpful here.

          Might make some changes to the patch to get this in ASAP.

          Show
          Mahadev konar added a comment - Eugene, We should write some unit tests for this. I am fine checking this into 3.4 for now. Can you please create a ticket to add a unit test for this? Mockito would be very helpful here. Might make some changes to the patch to get this in ASAP.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/1958/
          -----------------------------------------------------------

          (Updated 2011-09-22 18:39:22.010877)

          Review request for zookeeper.

          Changes
          -------

          Add link to JIRA.

          Summary
          -------

          Currently, in Zookeeper trunk, there are two problems with Kerberos TGT renewal:

          1. TGTs obtained from a keytab are not refreshed periodically. They should be, just as those from ticket cache are refreshed.

          2. Ticket renewal should be retried if it fails. Ticket renewal might fail if two or more separate processes (different JVMs) running as the same user try to renew Kerberos credentials at the same time.

          This addresses bug ZOOKEEPER-1181.
          https://issues.apache.org/jira/browse/ZOOKEEPER-1181

          Diffs


          src/java/main/org/apache/zookeeper/Login.java de64d0d

          Diff: https://reviews.apache.org/r/1958/diff

          Testing
          -------

          Have tested this with a Kerberized HBase/Hadoop cluster on Amazon EC2. Tested with a short Kerberos ticket life (modprinc -maxlife "5 minutes") for zookeeper server and clients. Tested with zookeeper server using a keytab and zookeeper client with ticket cache. Ran YCSB on HBase successfully on a one master, 3 regionserver cluster, where the master and 2 of the regionservers ran Quorum Peers.

          Thanks,

          Eugene

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1958/ ----------------------------------------------------------- (Updated 2011-09-22 18:39:22.010877) Review request for zookeeper. Changes ------- Add link to JIRA. Summary ------- Currently, in Zookeeper trunk, there are two problems with Kerberos TGT renewal: 1. TGTs obtained from a keytab are not refreshed periodically. They should be, just as those from ticket cache are refreshed. 2. Ticket renewal should be retried if it fails. Ticket renewal might fail if two or more separate processes (different JVMs) running as the same user try to renew Kerberos credentials at the same time. This addresses bug ZOOKEEPER-1181 . https://issues.apache.org/jira/browse/ZOOKEEPER-1181 Diffs src/java/main/org/apache/zookeeper/Login.java de64d0d Diff: https://reviews.apache.org/r/1958/diff Testing ------- Have tested this with a Kerberized HBase/Hadoop cluster on Amazon EC2. Tested with a short Kerberos ticket life (modprinc -maxlife "5 minutes") for zookeeper server and clients. Tested with zookeeper server using a keytab and zookeeper client with ticket cache. Ran YCSB on HBase successfully on a one master, 3 regionserver cluster, where the master and 2 of the regionservers ran Quorum Peers. Thanks, Eugene
          Hide
          Eugene Koontz added a comment -

          Hi Patrick,
          I'd like to push for this to be included in 3.4.0. It fixes significant problems with the currently-available Kerberos support in the 3.4.0 branch and trunk.
          Thanks for considering it,
          Eugene

          Show
          Eugene Koontz added a comment - Hi Patrick, I'd like to push for this to be included in 3.4.0. It fixes significant problems with the currently-available Kerberos support in the 3.4.0 branch and trunk. Thanks for considering it, Eugene
          Hide
          Eugene Koontz added a comment -

          Hi Thomas, Thanks for your interest! Please see the review here:

          https://reviews.apache.org/r/1958

          -Eugene

          Show
          Eugene Koontz added a comment - Hi Thomas, Thanks for your interest! Please see the review here: https://reviews.apache.org/r/1958 -Eugene
          Hide
          Thomas Koch added a comment -

          Could you please upload the patch to https://reviews.apache.org/r/new/ for review?

          Show
          Thomas Koch added a comment - Could you please upload the patch to https://reviews.apache.org/r/new/ for review?
          Hide
          Eugene Koontz added a comment -

          This was a one master, 3 regionserver cluster, where the master and 2 of the regionservers ran Quorum Peers.

          Show
          Eugene Koontz added a comment - This was a one master, 3 regionserver cluster, where the master and 2 of the regionservers ran Quorum Peers.
          Hide
          Eugene Koontz added a comment -

          -1 tests included. The patch doesn't appear to include any new or modified tests.

          Please justify why no new tests are needed for this patch.

          Also please list what manual steps were performed to verify this patch.

          Have tested this with a Kerberized HBase/Hadoop cluster on Amazon EC2. Tested with a short Kerberos ticket life (modprinc -maxlife "5 minutes") for zookeeper server and clients. Tested with zookeeper server using a keytab and zookeeper client with ticket cache. Ran YCSB on HBase successfully.

          I think I might be able to learn Mockito and mock up a Kerberos server for adding additional tests, but would rather defer that to later.

          Show
          Eugene Koontz added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. Have tested this with a Kerberized HBase/Hadoop cluster on Amazon EC2. Tested with a short Kerberos ticket life (modprinc -maxlife "5 minutes") for zookeeper server and clients. Tested with zookeeper server using a keytab and zookeeper client with ticket cache. Ran YCSB on HBase successfully. I think I might be able to learn Mockito and mock up a Kerberos server for adding additional tests, but would rather defer that to later.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12494526/ZOOKEEPER-1181.patch
          against trunk revision 1170458.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/538//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/538//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/538//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494526/ZOOKEEPER-1181.patch against trunk revision 1170458. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/538//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/538//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/538//console This message is automatically generated.
          Hide
          Patrick Hunt added a comment -

          also qa bot found some findbugs issues. Canceling for now.

          Show
          Patrick Hunt added a comment - also qa bot found some findbugs issues. Canceling for now.
          Hide
          Camille Fournier added a comment -

          I wish this was a touch cleaner... aren't reloginFromKeytab and reloginFromTicketCache almost the same method? Can we refactor the retry logic into one place?

          Show
          Camille Fournier added a comment - I wish this was a touch cleaner... aren't reloginFromKeytab and reloginFromTicketCache almost the same method? Can we refactor the retry logic into one place?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12494346/ZOOKEEPER-1181.patch
          against trunk revision 1170365.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/531//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/531//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/531//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494346/ZOOKEEPER-1181.patch against trunk revision 1170365. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/531//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/531//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/531//console This message is automatically generated.

            People

            • Assignee:
              Eugene Koontz
              Reporter:
              Eugene Koontz
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development