Here's a patch that adds some tests of KerberosUtil and fixes an issue where it will fail to perform the Kerberos auth because SIMPLE auth has already taken place with the OS username.

It looked like Kerberos would work if you allowed Flume to connect rather than having the user already logged in from the environment. Can you clarify the conditions where Kerberos doesn't work so we can gauge the severity of this bug?

The user is not already logged in, at least not with Kerberos. The issue is that:

      currentUser = UserGroupInformation.getLoginUser();

Will always return a UGI. When there is not TGT, it returns the current OS user with authentication method SIMPLE. That means this:

      if (finalPrincipal == null ||
          finalPrincipal.equals(currentUser.getUserName())) {

will always fail unless your username and the Kerberos principal match exactly. But in that case, there will be no TGT and you'll get GSS initiate failures later on.

It's possible this is caused by a change in Hadoop at some point, but the unit tests I added fail without the changes to KerberosUtil so I'd say it just plain doesn't work.

That explains why the check is finalPrincipal.equals(currentUser.getUserName(). Should this be a more sophisticated validation so that it works for both SIMPLE auth from the OS user and a properly logged-in user from Kerberos?

In the patch I added a check that the current user has Kerberos credentials:

if (currentUser != null && currentUser.hasKerberosCredentials())

before we compare the current user to the final principal. If there are no kerberos credentials, we always want to do a login from the keytab. The only case where we don't is if the principal and keytab are null (not set in the flume.conf) which I added a check for at the beginning of the login() method.

I also included a test for that case:

  @Test
  public void testNullLogin() throws IOException {
    String principal = null;
    String keytab = null;
    UserGroupInformation expResult = UserGroupInformation.getCurrentUser();
    UserGroupInformation result = KerberosUtil.login(principal, keytab);
    assertEquals(expResult, result);
  }

This doesn't test the case where the current user does have a TGT but wants to log in as a different user. I believe that's the case when another thread has already logged in and has a TGT. So we should exercise the finalPrincipal.equals(currentUser.getUserName() check in both the case where the final principal is already logged in and has a TGT and the case where we're trying to log in as a different user. Other than that, this looks good to me.

I think testFlumeLogin covers those but needs more commetns to make it clear:

  @Test
  public void testFlumeLogin() throws IOException {
    String principal = flumePrincipal;
    String keytab = flumeKeytab.getAbsolutePath();
    String expResult = principal;

    // Ensure login works when we don't already have a TGT
    String result = KerberosUtil.login(principal, keytab).getUserName();
    assertEquals("Initial login failed", expResult, result);

    // Ensure doing a second login with the same principal succeeds
    result = KerberosUtil.login(principal, keytab).getUserName();
    assertEquals("Re-login failed", expResult, result);

    // Make sure logging in with a new principal fails
    principal = alicePrincipal;
    keytab = aliceKeytab.getAbsolutePath();
    try {
      result = KerberosUtil.login(principal, keytab).getUserName();
      fail("Login should have failed with a new principal: " + result);
    } catch (KerberosUtil.SecurityException ex) {
      assertTrue("Login with a new principal failed, but for an unexpected "
          + "reason: " + ex.getMessage(),
          ex.getMessage().contains("Cannot use multiple Kerberos principals: "));
    }
  }

I added comments to call out the different scenerios above, and I can add those to the patch. I put all of these into a single test case as the login is JVM-wide and I'm not 100% certain how to control the order that tests are run, so I wanted them all in the same method.

+1 (non-binding)

Thanks for pointing out the tests and why they are in the same method. I missed it at first, but I agree that you have the scenarios I suggested covered.

+1

I think you guys know more about Kerberos than me, but the test passes and the patch LGTM. Committing this.

Thanks Joey for the patch, and thanks Ryan for the review!

Commit cec64a54abb8410a5e98cd2f815a5a839b49ce74 in flume's branch refs/heads/trunk from Mike Percy
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=cec64a5 ]

FLUME-2557. Make DatasetSink work with Kerberos

(Joey Echeverria via Mike Percy)

Commit d5c2cdac0a600355a8dd72b8c805e7dc3e5667be in flume's branch refs/heads/flume-1.6 from Mike Percy
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d5c2cda ]

FLUME-2557. Make DatasetSink work with Kerberos

(Joey Echeverria via Mike Percy)

SUCCESS: Integrated in flume-trunk #701 (See https://builds.apache.org/job/flume-trunk/701/)
FLUME-2557. Make DatasetSink work with Kerberos (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=cec64a54abb8410a5e98cd2f815a5a839b49ce74)

• flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/KerberosUtil.java
• flume-ng-sinks/flume-dataset-sink/src/test/java/org/apache/flume/sink/kite/TestKerberosUtil.java
• flume-ng-sinks/flume-dataset-sink/pom.xml
• flume-ng-sinks/flume-dataset-sink/src/test/resources/enable-kerberos.xml

SUCCESS: Integrated in Flume-trunk-hbase-98 #58 (See https://builds.apache.org/job/Flume-trunk-hbase-98/58/)
FLUME-2557. Make DatasetSink work with Kerberos (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=cec64a54abb8410a5e98cd2f815a5a839b49ce74)

• flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/KerberosUtil.java
• flume-ng-sinks/flume-dataset-sink/pom.xml
• flume-ng-sinks/flume-dataset-sink/src/test/java/org/apache/flume/sink/kite/TestKerberosUtil.java
• flume-ng-sinks/flume-dataset-sink/src/test/resources/enable-kerberos.xml