ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-1469

Adding Cross-Realm support for secure Zookeeper client authentication

    Details

    • Type: Improvement Improvement
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.4.3
    • Fix Version/s: 3.5.0
    • Component/s: documentation
    • Labels:
      None

      Description

      There is a use case where one needs to support cross realm authentication for zookeeper cluster. One use case is HBase Replication: HBase supports replicating data to multiple slave clusters, where the later might be running in different realms. With current zookeeper security, the region server of master HBase cluster are not able to query the zookeeper quorum members of the slave cluster. This jira is about adding such Xrealm support.

        Issue Links

          Activity

          Hide
          Himanshu Vashishtha added a comment -

          Let's say we have two REALMs: ABC.COM, and XYZ.COM. To enable Xrealm authentication, I added principals krbtgt/ABC.COM@XYZ.COM and krbtgt/XYZ.COM@ABC.COM with -require_preauth attribute, on both the clusters. Apart from that, I needed to modify the zookeeper principal to have -require_preauth attribute as it was giving a NO PREAUTH error:

          May 19 14:36:46 c1230.hal.cloudera.com krb5kdc[21238](info): TGS_REQ (5 etypes {3 1 23 16 17}) 172.29.81.100: NO PREAUTH: authtime 0,  hbase/c0318.hal.cloudera.com@CLOUDERA.COM for zookeeper/c1230.hal.cloudera.com@HAL.CLOUDERA.COM, Generic error (see e-text)
          

          I wonder whether this is the right approach, safe or unsafe? Please not that for HBase replication use case, there can be many to many relation... one cluster replicating data to multiple clusters and vice versa.

          After enabling Xrealm, I get the following exception:

          2012-05-19 22:47:26,529 [myid:] - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@137] - Failed to set name based on Kerberos authentication rules.
          

          This is because of the difference in the realm of client and server, and the RULE is set to DEFAULT: In the SaslServerCallbackHandler->handleAuthorizeCallback, kerberosName.getShortName() throws an IOException.

          Show
          Himanshu Vashishtha added a comment - Let's say we have two REALMs: ABC.COM, and XYZ.COM. To enable Xrealm authentication, I added principals krbtgt/ABC.COM@XYZ.COM and krbtgt/XYZ.COM@ABC.COM with -require_preauth attribute, on both the clusters. Apart from that, I needed to modify the zookeeper principal to have -require_preauth attribute as it was giving a NO PREAUTH error: May 19 14:36:46 c1230.hal.cloudera.com krb5kdc[21238](info): TGS_REQ (5 etypes {3 1 23 16 17}) 172.29.81.100: NO PREAUTH: authtime 0, hbase/c0318.hal.cloudera.com@CLOUDERA.COM for zookeeper/c1230.hal.cloudera.com@HAL.CLOUDERA.COM, Generic error (see e-text) I wonder whether this is the right approach, safe or unsafe? Please not that for HBase replication use case, there can be many to many relation... one cluster replicating data to multiple clusters and vice versa. After enabling Xrealm, I get the following exception: 2012-05-19 22:47:26,529 [myid:] - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@137] - Failed to set name based on Kerberos authentication rules. This is because of the difference in the realm of client and server, and the RULE is set to DEFAULT: In the SaslServerCallbackHandler->handleAuthorizeCallback, kerberosName.getShortName() throws an IOException.
          Hide
          Andrew Purtell added a comment -

          I think this issue should be an improvement, not a bug. The proposal extends current functionality to handle additional use cases not scoped for the initial work.

          Show
          Andrew Purtell added a comment - I think this issue should be an improvement, not a bug. The proposal extends current functionality to handle additional use cases not scoped for the initial work.
          Hide
          Patrick Hunt added a comment -

          ZooKeeper supports rule specification for short name mapping similar to Hadoop via "zookeeper.security.auth_to_local" - I'm wondering if we could take advantage of that here?

          Show
          Patrick Hunt added a comment - ZooKeeper supports rule specification for short name mapping similar to Hadoop via "zookeeper.security.auth_to_local" - I'm wondering if we could take advantage of that here?
          Hide
          Eugene Koontz added a comment -

          Hi Himanshu, I'm adding a patch to this JIRA to print out the exception that was thrown in your log statement above; could you try applying this patch and let me know what addition information is shown in your testing scenario?

          Thanks, Eugene

          Show
          Eugene Koontz added a comment - Hi Himanshu, I'm adding a patch to this JIRA to print out the exception that was thrown in your log statement above; could you try applying this patch and let me know what addition information is shown in your testing scenario? Thanks, Eugene
          Hide
          Eugene Koontz added a comment -

          print exception thrown from kerberosName.getShortName().

          Show
          Eugene Koontz added a comment - print exception thrown from kerberosName.getShortName().
          Hide
          Himanshu Vashishtha added a comment -

          The exception is:

          2012-05-21 16:35:11,283 [myid:] - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@137] - Failed to set name based on Kerberos authentication rules.org.apache.zookeeper.server.auth.KerberosName$NoMatchingRule: No rules applied to hbase/c0319.hal.cloudera.com@CLOUDERA.COM
          
          
          Show
          Himanshu Vashishtha added a comment - The exception is: 2012-05-21 16:35:11,283 [myid:] - ERROR [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@137] - Failed to set name based on Kerberos authentication rules.org.apache.zookeeper.server.auth.KerberosName$NoMatchingRule: No rules applied to hbase/c0319.hal.cloudera.com@CLOUDERA.COM
          Hide
          Eugene Koontz added a comment -

          Thanks Himanshu - can you try setting zookeeper.security.auth_to_local to [2:$1] - see : https://ccp.cloudera.com/display/CDHDOC/Appendix+C+-+Configuring+the+Mapping+from+Kerberos+Principals+to+Short+Names

          -Eugene

          Show
          Eugene Koontz added a comment - Thanks Himanshu - can you try setting zookeeper.security.auth_to_local to [2:$1] - see : https://ccp.cloudera.com/display/CDHDOC/Appendix+C+-+Configuring+the+Mapping+from+Kerberos+Principals+to+Short+Names -Eugene
          Hide
          Himanshu Vashishtha added a comment -

          Thanks Eugene.
          Adding rule for that realm worked, and I am able to connect a Regionserver of a slave cluster (which means request 'ls /hbase/rs') was served successfully.

          @Eugene: It will be good to know the repercussions for setting -requires_preauth attribute to the zookeeper principal. Without this change, it was getting No PREAUTH error in the kerberos logs.

          Show
          Himanshu Vashishtha added a comment - Thanks Eugene. Adding rule for that realm worked, and I am able to connect a Regionserver of a slave cluster (which means request 'ls /hbase/rs') was served successfully. @Eugene: It will be good to know the repercussions for setting -requires_preauth attribute to the zookeeper principal. Without this change, it was getting No PREAUTH error in the kerberos logs.
          Hide
          Eugene Koontz added a comment -

          Hey, great to hear that! Patrick deserves the credit for suggesting setting auth_to_local.

          What shall we do with this JIRA now?

          Show
          Eugene Koontz added a comment - Hey, great to hear that! Patrick deserves the credit for suggesting setting auth_to_local. What shall we do with this JIRA now?
          Hide
          Eugene Koontz added a comment -

          With regard to -requires_preauth, I need to study it further and try to reproduce your secure HBase replication setup. We should document within the HBase docs what you've accomplished - perhaps an HBase JIRA is warranted similar to https://issues.apache.org/jira/browse/HBASE-4960.

          Show
          Eugene Koontz added a comment - With regard to -requires_preauth, I need to study it further and try to reproduce your secure HBase replication setup. We should document within the HBase docs what you've accomplished - perhaps an HBase JIRA is warranted similar to https://issues.apache.org/jira/browse/HBASE-4960 .
          Hide
          Himanshu Vashishtha added a comment -

          I enabled the cross realm hbase replication after adding rules for zookeeper and hadoop.
          So, the steps are:

          • Add tgt principals for cross realm: add principals krbtgt/FIRST.COM@SECOND.COM and krbtgt/SECOND.COM@FIRST.COM, in both the realms.
          • Add rules in the slave zookeeper quorum to let it create the short names based on the incoming principal, using the system property: -Dzookeeper.security.auth_to_local=RULE:[2:\$1@\$0](.*@\\QFIRST.COM\\E$)s/@\\QFIRST.COM
            E$//DEFAULT
          • Add rules in the core-site.xml of the slave cluster hadoop setup:
            <property>
              <name>hadoop.security.auth_to_local</name>
                <value>
                RULE:[2:$1@$0](.*@\QFIRST.COM\E$)s/@\QFIRST.COM\E$//
                DEFAULT
                </value> 
            

          The above rules are for principals which have both service and instance in them (service/instance@REALM).

          Regarding -requires_preauth, its documented at the mit docs. But then when I used that, I was getting errors to do the same for zookeeper, and hadoop principals too. So, I went ahead with the default ones (which requires pre_auth). Closing out this jira now.

          Thanks to Eugene and Patrick.

          Show
          Himanshu Vashishtha added a comment - I enabled the cross realm hbase replication after adding rules for zookeeper and hadoop. So, the steps are: Add tgt principals for cross realm: add principals krbtgt/FIRST.COM@SECOND.COM and krbtgt/SECOND.COM@FIRST.COM, in both the realms. Add rules in the slave zookeeper quorum to let it create the short names based on the incoming principal, using the system property: -Dzookeeper.security.auth_to_local=RULE: [2:\$1@\$0] (.*@\\QFIRST.COM\\E$)s/@\\QFIRST.COM E$//DEFAULT Add rules in the core-site.xml of the slave cluster hadoop setup: <property> <name>hadoop.security.auth_to_local</name> <value> RULE:[2:$1@$0](.*@\QFIRST.COM\E$)s/@\QFIRST.COM\E$ // DEFAULT </value> The above rules are for principals which have both service and instance in them (service/instance@REALM). Regarding -requires_preauth, its documented at the mit docs. But then when I used that, I was getting errors to do the same for zookeeper, and hadoop principals too. So, I went ahead with the default ones (which requires pre_auth). Closing out this jira now. Thanks to Eugene and Patrick.
          Hide
          Patrick Hunt added a comment -

          We should document this. Eugene can you do this? thx.

          Show
          Patrick Hunt added a comment - We should document this. Eugene can you do this? thx.
          Hide
          Eugene Koontz added a comment -

          No problem to document this (this would be in the cwiki, I assume, like https://cwiki.apache.org/ZOOKEEPER/zookeeper-and-sasl.html).

          Thanks Himanshu for the writeup of your working configuration!

          Show
          Eugene Koontz added a comment - No problem to document this (this would be in the cwiki, I assume, like https://cwiki.apache.org/ZOOKEEPER/zookeeper-and-sasl.html ). Thanks Himanshu for the writeup of your working configuration!
          Hide
          Patrick Hunt added a comment -

          Yes, seems like updating the cwiki with a new section would work.

          Show
          Patrick Hunt added a comment - Yes, seems like updating the cwiki with a new section would work.
          Hide
          Mahadev konar added a comment -

          Eugene,
          Looks like you forgot to add to the wiki. Can you please do that? We can go ahead and close this jira then.

          thanks

          Show
          Mahadev konar added a comment - Eugene, Looks like you forgot to add to the wiki. Can you please do that? We can go ahead and close this jira then. thanks
          Hide
          Mahadev konar added a comment -

          Moving it out of 3.4 release.

          Show
          Mahadev konar added a comment - Moving it out of 3.4 release.

            People

            • Assignee:
              Eugene Koontz
              Reporter:
              Himanshu Vashishtha
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development