Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2096

Document necessary configuration for Kerberos with master CNAMEs

    Details

    • Type: Task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: documentation, security
    • Labels:
      None
    • Target Version/s:

      Description

      Currently our docs recommend using CNAMEs for master addresses to simplify moving them around. However, if clients connect to a master with its non-canonical name, there are some complications with Kerberos principals, etc. We should test and document the necessary steps for such a configuration.

        Issue Links

          Activity

          Hide
          tlipcon Todd Lipcon added a comment -

          A few notes:

          • the C++ client (at least in Kudu 1.4 and earlier) always resolves the passed host name into an IP and passes that into the SASL and krb5 libraries. We then rely on 'rdns = true' setting (the default) in krb5.conf to resolve it back to the canonical name of the host. So, if there is a DNS configuration like:
          kudumaster.example.com CNAME server1.example.com
          server1.example.com A 1.2.3.4
          4.3.2.1.in-addr.arpa. PTR server1.example.com
          

          then the C++ client will resolve 'kudumaster.example.com' to '1.2.3.4', and then krb5 will reverse it to 'server1.example.com', the FQDN of the actual host. The server already assumes and requires that its keytab have a principal kudu/FQDN@REALM, so the incoming client's expected principal will match the actual principal used by the server.

          The Java client on the other hand seems to skip the canonicalization and will try to find a principal kudu/kudumaster.example.com and fail, since the correct principal is kudu/server1.example.com.

          Show
          tlipcon Todd Lipcon added a comment - A few notes: the C++ client (at least in Kudu 1.4 and earlier) always resolves the passed host name into an IP and passes that into the SASL and krb5 libraries. We then rely on 'rdns = true' setting (the default) in krb5.conf to resolve it back to the canonical name of the host. So, if there is a DNS configuration like: kudumaster.example.com CNAME server1.example.com server1.example.com A 1.2.3.4 4.3.2.1.in-addr.arpa. PTR server1.example.com then the C++ client will resolve 'kudumaster.example.com' to '1.2.3.4', and then krb5 will reverse it to 'server1.example.com', the FQDN of the actual host. The server already assumes and requires that its keytab have a principal kudu/FQDN@REALM, so the incoming client's expected principal will match the actual principal used by the server. The Java client on the other hand seems to skip the canonicalization and will try to find a principal kudu/kudumaster.example.com and fail, since the correct principal is kudu/server1.example.com.
          Hide
          tlipcon Todd Lipcon added a comment -

          It appears the JDK Kerberos implementation does some limited canonicalization: https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/sun/security/krb5/PrincipalName.java#L386

          Specifically, it calls InetAddress.getByName and if the resulting name is just more-qualified version of the original name ("foo" -> "foo.example.com") then it will do that canonicalization.

          It seems that if we are OK using some internal APIs, we can use sun.security.krb5.Config to read the configured krb5.conf and match the behavior of the C++ client without having to add a new Java-specific configuration in the client.

          Show
          tlipcon Todd Lipcon added a comment - It appears the JDK Kerberos implementation does some limited canonicalization: https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/sun/security/krb5/PrincipalName.java#L386 Specifically, it calls InetAddress.getByName and if the resulting name is just more-qualified version of the original name ("foo" -> "foo.example.com") then it will do that canonicalization. It seems that if we are OK using some internal APIs, we can use sun.security.krb5.Config to read the configured krb5.conf and match the behavior of the C++ client without having to add a new Java-specific configuration in the client.
          Hide
          r1pp3rj4ck Attila Bukor added a comment -

          Todd Lipcon this approach wouldn't work if the user has CNAME-s for the host like we suggest in the docs, e.g. the way you used in your previous example, right?

          kudumaster.example.com CNAME server1.example.com
          server1.example.com A 1.2.3.4
          4.3.2.1.in-addr.arpa. PTR server1.example.com
          
          Show
          r1pp3rj4ck Attila Bukor added a comment - Todd Lipcon this approach wouldn't work if the user has CNAME-s for the host like we suggest in the docs, e.g. the way you used in your previous example, right? kudumaster.example.com CNAME server1.example.com server1.example.com A 1.2.3.4 4.3.2.1.in-addr.arpa. PTR server1.example.com
          Hide
          tlipcon Todd Lipcon added a comment -

          If you check out the chart in KUDU-2032, you can actually see the behavior that MIT krb5 has for different settings for an example of a CNAME. I think the defaults (canonicalize_host_name = true, rdns = true) would work as expected for the CNAME config – it would resolve all the way to an IP, then reverse back to the true FQDN of the host, and use that as a principal. The trick is that it's possible to configure krb5 to do only the "cname -> actual name" step, which might actually not match the reversed IP.

          Show
          tlipcon Todd Lipcon added a comment - If you check out the chart in KUDU-2032 , you can actually see the behavior that MIT krb5 has for different settings for an example of a CNAME. I think the defaults (canonicalize_host_name = true, rdns = true) would work as expected for the CNAME config – it would resolve all the way to an IP, then reverse back to the true FQDN of the host, and use that as a principal. The trick is that it's possible to configure krb5 to do only the "cname -> actual name" step, which might actually not match the reversed IP.
          Hide
          r1pp3rj4ck Attila Bukor added a comment -

          So we're storing each master address as a Guava com.google.common.net.HostAndPort which stores it internally as a String. This is then passed to com.sun.security.sasl.gsskerb.GssKrb5Client unmodified currently which also doesn't try to resolve it or canonicalize it no matter what's configured in /etc/krb5.conf if I'm reading it correctly and I assume this is the problem.

          So another problem is I couldn't find a way to mimic the canon == true && rdns == false way (at least not without some DNS library) as from the String we need to create an InetAddress by calling InetAddress.getByName(String host) which would result in reverse resolution by default. After this we can call InetAddress.getCanonicalHostName() on it depending if canonicalize_host_name is set true. In case it's false we can skip this whole thing and return the original String. This way we can still control this behavior with the canonicalize_host_name flag, but the default is also sensible, the only thing we can't do is basically the one that doesn't really make sense.

          Do you think this is a good approach? Should I open a bug/improvement JIRA for this or should I just reference this one in the commit? I have a working proof-of-concept, I need to make it configurable based on your decision and some tests.

          Show
          r1pp3rj4ck Attila Bukor added a comment - So we're storing each master address as a Guava com.google.common.net.HostAndPort which stores it internally as a String. This is then passed to com.sun.security.sasl.gsskerb.GssKrb5Client unmodified currently which also doesn't try to resolve it or canonicalize it no matter what's configured in /etc/krb5.conf if I'm reading it correctly and I assume this is the problem. So another problem is I couldn't find a way to mimic the canon == true && rdns == false way (at least not without some DNS library) as from the String we need to create an InetAddress by calling InetAddress.getByName(String host) which would result in reverse resolution by default. After this we can call InetAddress.getCanonicalHostName() on it depending if canonicalize_host_name is set true. In case it's false we can skip this whole thing and return the original String. This way we can still control this behavior with the canonicalize_host_name flag, but the default is also sensible, the only thing we can't do is basically the one that doesn't really make sense. Do you think this is a good approach? Should I open a bug/improvement JIRA for this or should I just reference this one in the commit? I have a working proof-of-concept, I need to make it configurable based on your decision and some tests.
          Hide
          tlipcon Todd Lipcon added a comment -

          I see your point that it is probably not possible to duplicate the 'canon = true & rdns = false' configuration in Java. That's a shame since it means that C++ and Java clients may act different on those systems.

          I spent some time looking at the Hadoop RPC code, since I figure that many people installing Kudu (particularly with the Java client) probably already have working systems with Hadoop's Kerberos usage. It seems it is quite configurable, but more or less does something like:

          • the server actually advertises its own principal based on what it thinks its FQDN is, during RPC negotiation, before the SASL negotiation starts
          • the client then validates it using a couple rules:
            • the user may specify a regex pattern of what is allowed ( see HADOOP-9789 ). This isn't often used as far as I know.
            • if there is no regex, and assuming no other exotic configs set up (there is a 'use_ip' config which disables any canonicalization if the user specifies an IP), it ends up calling to InetAddress.getCanonicalHostName (in SecurityUtil#getServerPrincipal(String, InetAddress)

          We've generally required that Kudu clusters have sane DNS where every host has a canonical name and IP which have correct forward and reverse lookups. So I think probably the best option here is to just change the Java client to always use getCanonicalHostName() rather than listening to whatever host is specified by the user. If we need to add some complicated configurations later we can consider it, but I think this should suffice for 99% of correctly-set-up clusters even if it doesn't perfectly respect krb5.conf.

          Show
          tlipcon Todd Lipcon added a comment - I see your point that it is probably not possible to duplicate the 'canon = true & rdns = false' configuration in Java. That's a shame since it means that C++ and Java clients may act different on those systems. I spent some time looking at the Hadoop RPC code, since I figure that many people installing Kudu (particularly with the Java client) probably already have working systems with Hadoop's Kerberos usage. It seems it is quite configurable, but more or less does something like: the server actually advertises its own principal based on what it thinks its FQDN is, during RPC negotiation, before the SASL negotiation starts the client then validates it using a couple rules: the user may specify a regex pattern of what is allowed ( see HADOOP-9789 ). This isn't often used as far as I know. if there is no regex, and assuming no other exotic configs set up (there is a 'use_ip' config which disables any canonicalization if the user specifies an IP), it ends up calling to InetAddress.getCanonicalHostName (in SecurityUtil#getServerPrincipal(String, InetAddress) We've generally required that Kudu clusters have sane DNS where every host has a canonical name and IP which have correct forward and reverse lookups. So I think probably the best option here is to just change the Java client to always use getCanonicalHostName() rather than listening to whatever host is specified by the user. If we need to add some complicated configurations later we can consider it, but I think this should suffice for 99% of correctly-set-up clusters even if it doesn't perfectly respect krb5.conf.
          Hide
          r1pp3rj4ck Attila Bukor added a comment -

          Todd Lipcon okay, I can likely push it to gerrit today, I just want to verify the existing tests run locally (need to build the native libs as my build is very old) and I'll try to figure out how to test this patch. Should I open a new JIRA for it or is this one fine?

          Show
          r1pp3rj4ck Attila Bukor added a comment - Todd Lipcon okay, I can likely push it to gerrit today, I just want to verify the existing tests run locally (need to build the native libs as my build is very old) and I'll try to figure out how to test this patch. Should I open a new JIRA for it or is this one fine?
          Hide
          tlipcon Todd Lipcon added a comment -

          Let's open a new JIRA like you suggested, specifically focusing on ensuring that the Java client canonicalizes hostnames for use in kerberos principals.

          BTW I think we should also be lower-casing the hostname after canonicalization (see KUDU-1942)

          Show
          tlipcon Todd Lipcon added a comment - Let's open a new JIRA like you suggested, specifically focusing on ensuring that the Java client canonicalizes hostnames for use in kerberos principals. BTW I think we should also be lower-casing the hostname after canonicalization (see KUDU-1942 )
          Hide
          r1pp3rj4ck Attila Bukor added a comment -

          Todd Lipcon what's the plan with this JIRA now that KUDU-2103 is fixed and merged in?

          Show
          r1pp3rj4ck Attila Bukor added a comment - Todd Lipcon what's the plan with this JIRA now that KUDU-2103 is fixed and merged in?
          Hide
          tlipcon Todd Lipcon added a comment -

          I think we should set up such a cluster wth Kerberos on, with the master using a CNAME and do a couple end-to-end tests including Impala configured to connect via the CNAME. Since Impala uses both the C++ and Java clients I think this should be pretty good assurance that the whole thing is working as expected. Let me know if you are interested in helping with this – otherwise I can try to find time to do it in the several weeks (but maybe not in time for the 1.5 release itself).

          Show
          tlipcon Todd Lipcon added a comment - I think we should set up such a cluster wth Kerberos on, with the master using a CNAME and do a couple end-to-end tests including Impala configured to connect via the CNAME. Since Impala uses both the C++ and Java clients I think this should be pretty good assurance that the whole thing is working as expected. Let me know if you are interested in helping with this – otherwise I can try to find time to do it in the several weeks (but maybe not in time for the 1.5 release itself).
          Hide
          r1pp3rj4ck Attila Bukor added a comment -

          Todd Lipcon I'm definitely interested, I'll set up a test for this.

          Show
          r1pp3rj4ck Attila Bukor added a comment - Todd Lipcon I'm definitely interested, I'll set up a test for this.
          Hide
          r1pp3rj4ck Attila Bukor added a comment -

          It seems this is working fine from Impala also if it's using the correct kudu-client JAR:

          > create table test (id bigint, name string, primary key (id)) partition by hash partitions 2 stored as kudu tblproperties ('kudu.master_addresses' = 'kudu-master');
          Query: create table test (id bigint, name string, primary key (id)) partition by hash partitions 2 stored as kudu tblproperties ('kudu.master_addresses' = 'kudu-master')
          Fetched 0 row(s) in 0.85s
          
          Show
          r1pp3rj4ck Attila Bukor added a comment - It seems this is working fine from Impala also if it's using the correct kudu-client JAR: > create table test (id bigint, name string, primary key (id)) partition by hash partitions 2 stored as kudu tblproperties ('kudu.master_addresses' = 'kudu-master'); Query: create table test (id bigint, name string, primary key (id)) partition by hash partitions 2 stored as kudu tblproperties ('kudu.master_addresses' = 'kudu-master') Fetched 0 row(s) in 0.85s

            People

            • Assignee:
              Unassigned
              Reporter:
              tlipcon Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development