Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2557

DatasetSink doesn't work with Kerberos

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.2
    • Fix Version/s: 1.6.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      When configuring Kerberos, you'll get an error such as:

      Sink kite-dataset has been removed due to an error during configuration
      org.apache.flume.sink.kite.KerberosUtil$SecurityException: Cannot use
      multiple Kerberos principals:
      flume/joey-book-3.ent.cloudera.com@EXAMPLE.COM would replace flume

      1. FLUME-2557.patch
        9 kB
        Joey Echeverria

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Flume-trunk-hbase-98 #58 (See https://builds.apache.org/job/Flume-trunk-hbase-98/58/)
        FLUME-2557. Make DatasetSink work with Kerberos (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=cec64a54abb8410a5e98cd2f815a5a839b49ce74)

        • flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/KerberosUtil.java
        • flume-ng-sinks/flume-dataset-sink/pom.xml
        • flume-ng-sinks/flume-dataset-sink/src/test/java/org/apache/flume/sink/kite/TestKerberosUtil.java
        • flume-ng-sinks/flume-dataset-sink/src/test/resources/enable-kerberos.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-98 #58 (See https://builds.apache.org/job/Flume-trunk-hbase-98/58/ ) FLUME-2557 . Make DatasetSink work with Kerberos (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=cec64a54abb8410a5e98cd2f815a5a839b49ce74 ) flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/KerberosUtil.java flume-ng-sinks/flume-dataset-sink/pom.xml flume-ng-sinks/flume-dataset-sink/src/test/java/org/apache/flume/sink/kite/TestKerberosUtil.java flume-ng-sinks/flume-dataset-sink/src/test/resources/enable-kerberos.xml
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in flume-trunk #701 (See https://builds.apache.org/job/flume-trunk/701/)
        FLUME-2557. Make DatasetSink work with Kerberos (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=cec64a54abb8410a5e98cd2f815a5a839b49ce74)

        • flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/KerberosUtil.java
        • flume-ng-sinks/flume-dataset-sink/src/test/java/org/apache/flume/sink/kite/TestKerberosUtil.java
        • flume-ng-sinks/flume-dataset-sink/pom.xml
        • flume-ng-sinks/flume-dataset-sink/src/test/resources/enable-kerberos.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #701 (See https://builds.apache.org/job/flume-trunk/701/ ) FLUME-2557 . Make DatasetSink work with Kerberos (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=cec64a54abb8410a5e98cd2f815a5a839b49ce74 ) flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/KerberosUtil.java flume-ng-sinks/flume-dataset-sink/src/test/java/org/apache/flume/sink/kite/TestKerberosUtil.java flume-ng-sinks/flume-dataset-sink/pom.xml flume-ng-sinks/flume-dataset-sink/src/test/resources/enable-kerberos.xml
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit d5c2cdac0a600355a8dd72b8c805e7dc3e5667be in flume's branch refs/heads/flume-1.6 from Mike Percy
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d5c2cda ]

        FLUME-2557. Make DatasetSink work with Kerberos

        (Joey Echeverria via Mike Percy)

        Show
        jira-bot ASF subversion and git services added a comment - Commit d5c2cdac0a600355a8dd72b8c805e7dc3e5667be in flume's branch refs/heads/flume-1.6 from Mike Percy [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d5c2cda ] FLUME-2557 . Make DatasetSink work with Kerberos (Joey Echeverria via Mike Percy)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit cec64a54abb8410a5e98cd2f815a5a839b49ce74 in flume's branch refs/heads/trunk from Mike Percy
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=cec64a5 ]

        FLUME-2557. Make DatasetSink work with Kerberos

        (Joey Echeverria via Mike Percy)

        Show
        jira-bot ASF subversion and git services added a comment - Commit cec64a54abb8410a5e98cd2f815a5a839b49ce74 in flume's branch refs/heads/trunk from Mike Percy [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=cec64a5 ] FLUME-2557 . Make DatasetSink work with Kerberos (Joey Echeverria via Mike Percy)
        Hide
        mpercy Mike Percy added a comment -

        +1

        I think you guys know more about Kerberos than me, but the test passes and the patch LGTM. Committing this.

        Thanks Joey for the patch, and thanks Ryan for the review!

        Show
        mpercy Mike Percy added a comment - +1 I think you guys know more about Kerberos than me, but the test passes and the patch LGTM. Committing this. Thanks Joey for the patch, and thanks Ryan for the review!
        Hide
        rdblue Ryan Blue added a comment - - edited

        +1 (non-binding)

        Thanks for pointing out the tests and why they are in the same method. I missed it at first, but I agree that you have the scenarios I suggested covered.

        Show
        rdblue Ryan Blue added a comment - - edited +1 (non-binding) Thanks for pointing out the tests and why they are in the same method. I missed it at first, but I agree that you have the scenarios I suggested covered.
        Hide
        fwiffo Joey Echeverria added a comment -

        I think testFlumeLogin covers those but needs more commetns to make it clear:

          @Test
          public void testFlumeLogin() throws IOException {
            String principal = flumePrincipal;
            String keytab = flumeKeytab.getAbsolutePath();
            String expResult = principal;
        
            // Ensure login works when we don't already have a TGT
            String result = KerberosUtil.login(principal, keytab).getUserName();
            assertEquals("Initial login failed", expResult, result);
        
            // Ensure doing a second login with the same principal succeeds
            result = KerberosUtil.login(principal, keytab).getUserName();
            assertEquals("Re-login failed", expResult, result);
        
            // Make sure logging in with a new principal fails
            principal = alicePrincipal;
            keytab = aliceKeytab.getAbsolutePath();
            try {
              result = KerberosUtil.login(principal, keytab).getUserName();
              fail("Login should have failed with a new principal: " + result);
            } catch (KerberosUtil.SecurityException ex) {
              assertTrue("Login with a new principal failed, but for an unexpected "
                  + "reason: " + ex.getMessage(),
                  ex.getMessage().contains("Cannot use multiple Kerberos principals: "));
            }
          }
        

        I added comments to call out the different scenerios above, and I can add those to the patch. I put all of these into a single test case as the login is JVM-wide and I'm not 100% certain how to control the order that tests are run, so I wanted them all in the same method.

        Show
        fwiffo Joey Echeverria added a comment - I think testFlumeLogin covers those but needs more commetns to make it clear: @Test public void testFlumeLogin() throws IOException { String principal = flumePrincipal; String keytab = flumeKeytab.getAbsolutePath(); String expResult = principal; // Ensure login works when we don't already have a TGT String result = KerberosUtil.login(principal, keytab).getUserName(); assertEquals( "Initial login failed" , expResult, result); // Ensure doing a second login with the same principal succeeds result = KerberosUtil.login(principal, keytab).getUserName(); assertEquals( "Re-login failed" , expResult, result); // Make sure logging in with a new principal fails principal = alicePrincipal; keytab = aliceKeytab.getAbsolutePath(); try { result = KerberosUtil.login(principal, keytab).getUserName(); fail( "Login should have failed with a new principal: " + result); } catch (KerberosUtil.SecurityException ex) { assertTrue( "Login with a new principal failed, but for an unexpected " + "reason: " + ex.getMessage(), ex.getMessage().contains( "Cannot use multiple Kerberos principals: " )); } } I added comments to call out the different scenerios above, and I can add those to the patch. I put all of these into a single test case as the login is JVM-wide and I'm not 100% certain how to control the order that tests are run, so I wanted them all in the same method.
        Hide
        rdblue Ryan Blue added a comment -

        This doesn't test the case where the current user does have a TGT but wants to log in as a different user. I believe that's the case when another thread has already logged in and has a TGT. So we should exercise the finalPrincipal.equals(currentUser.getUserName() check in both the case where the final principal is already logged in and has a TGT and the case where we're trying to log in as a different user. Other than that, this looks good to me.

        Show
        rdblue Ryan Blue added a comment - This doesn't test the case where the current user does have a TGT but wants to log in as a different user. I believe that's the case when another thread has already logged in and has a TGT. So we should exercise the finalPrincipal.equals(currentUser.getUserName() check in both the case where the final principal is already logged in and has a TGT and the case where we're trying to log in as a different user. Other than that, this looks good to me.
        Hide
        fwiffo Joey Echeverria added a comment -

        I also included a test for that case:

          @Test
          public void testNullLogin() throws IOException {
            String principal = null;
            String keytab = null;
            UserGroupInformation expResult = UserGroupInformation.getCurrentUser();
            UserGroupInformation result = KerberosUtil.login(principal, keytab);
            assertEquals(expResult, result);
          }
        
        Show
        fwiffo Joey Echeverria added a comment - I also included a test for that case: @Test public void testNullLogin() throws IOException { String principal = null ; String keytab = null ; UserGroupInformation expResult = UserGroupInformation.getCurrentUser(); UserGroupInformation result = KerberosUtil.login(principal, keytab); assertEquals(expResult, result); }
        Hide
        fwiffo Joey Echeverria added a comment -

        In the patch I added a check that the current user has Kerberos credentials:

        if (currentUser != null && currentUser.hasKerberosCredentials())
        

        before we compare the current user to the final principal. If there are no kerberos credentials, we always want to do a login from the keytab. The only case where we don't is if the principal and keytab are null (not set in the flume.conf) which I added a check for at the beginning of the login() method.

        Show
        fwiffo Joey Echeverria added a comment - In the patch I added a check that the current user has Kerberos credentials: if (currentUser != null && currentUser.hasKerberosCredentials()) before we compare the current user to the final principal. If there are no kerberos credentials, we always want to do a login from the keytab. The only case where we don't is if the principal and keytab are null (not set in the flume.conf) which I added a check for at the beginning of the login() method.
        Hide
        rdblue Ryan Blue added a comment -

        That explains why the check is finalPrincipal.equals(currentUser.getUserName(). Should this be a more sophisticated validation so that it works for both SIMPLE auth from the OS user and a properly logged-in user from Kerberos?

        Show
        rdblue Ryan Blue added a comment - That explains why the check is finalPrincipal.equals(currentUser.getUserName() . Should this be a more sophisticated validation so that it works for both SIMPLE auth from the OS user and a properly logged-in user from Kerberos?
        Hide
        fwiffo Joey Echeverria added a comment -

        The user is not already logged in, at least not with Kerberos. The issue is that:

              currentUser = UserGroupInformation.getLoginUser();
        

        Will always return a UGI. When there is not TGT, it returns the current OS user with authentication method SIMPLE. That means this:

              if (finalPrincipal == null ||
                  finalPrincipal.equals(currentUser.getUserName())) {
        

        will always fail unless your username and the Kerberos principal match exactly. But in that case, there will be no TGT and you'll get GSS initiate failures later on.

        It's possible this is caused by a change in Hadoop at some point, but the unit tests I added fail without the changes to KerberosUtil so I'd say it just plain doesn't work.

        Show
        fwiffo Joey Echeverria added a comment - The user is not already logged in, at least not with Kerberos. The issue is that: currentUser = UserGroupInformation.getLoginUser(); Will always return a UGI. When there is not TGT, it returns the current OS user with authentication method SIMPLE. That means this: if (finalPrincipal == null || finalPrincipal.equals(currentUser.getUserName())) { will always fail unless your username and the Kerberos principal match exactly. But in that case, there will be no TGT and you'll get GSS initiate failures later on. It's possible this is caused by a change in Hadoop at some point, but the unit tests I added fail without the changes to KerberosUtil so I'd say it just plain doesn't work.
        Hide
        rdblue Ryan Blue added a comment -

        It looked like Kerberos would work if you allowed Flume to connect rather than having the user already logged in from the environment. Can you clarify the conditions where Kerberos doesn't work so we can gauge the severity of this bug?

        Show
        rdblue Ryan Blue added a comment - It looked like Kerberos would work if you allowed Flume to connect rather than having the user already logged in from the environment. Can you clarify the conditions where Kerberos doesn't work so we can gauge the severity of this bug?
        Hide
        fwiffo Joey Echeverria added a comment -

        Here's a patch that adds some tests of KerberosUtil and fixes an issue where it will fail to perform the Kerberos auth because SIMPLE auth has already taken place with the OS username.

        Show
        fwiffo Joey Echeverria added a comment - Here's a patch that adds some tests of KerberosUtil and fixes an issue where it will fail to perform the Kerberos auth because SIMPLE auth has already taken place with the OS username.

          People

          • Assignee:
            fwiffo Joey Echeverria
            Reporter:
            fwiffo Joey Echeverria
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development