[SENTRY-1703] Solr-Sentry in kerberos mode makes too many KDC requests and returns unauthorized on KDC timeout

• Type: Bug
• Status: Open
• Priority: Blocker
• Resolution: Unresolved
• Affects Version/s: 1.5.1
• Component/s: Solr Plugin
• Assignee: Unassigned
• Reporter: Tushar I
• Created: 11/Apr/17 23:54
• Updated: 24/May/17 23:44

Description

Sentry Version: 1.5.1-cdh5.8.0

We are seeing intermittent authorization failures with Sentry Solr plugin in a Kerberos environment.

1. We are writing to Solr using the SolrJ client from within Spark jobs in a multi-node Spark/Hadoop cluster and frequently get authorization errors from Solr in individual spark tasks saying "User XX does not have privileges for YYcollection" which are generated by the Solr-Sentry plugin. (The user very well has access to the collection and it works fine rest of the times).
2. The root cause seems to be that on every Solr call from the client, Sentry reaches out to KDC on behalf of solr/hostname, thereby drowning the KDC in tons of requests per second, and at some point fails on a KDC timeout, throwing the exception: org.apache.sentry.binding.solr.authz.SentrySolrAuthorizationException: User XX does not have privileges for YYcollection to the calling client.

I didn't get enough time to investigate why Sentry is making so many KDC calls, maybe it's doing it for each document in a batched Solr operation, or it logs in using keytab each time and doesn't cache the ticket, etc.

Caching the result of authProvider.hasAccess() in SolrAuthzBinding.java for a reasonably short time might not be a bad idea.

My question in the meantime is: Are there any tuning knobs to somehow reduce the load on KDC, or increase the KDC request timeout value, or anything along these lines?

Relevant stacktraces captured from Solr Admin are attached:
1. stacktrace1.log : The timeout from KDC for sentry call
2. stacktrace2.log: When Sentry cannot authenticate with KDC due to # 1 above
3. stacktrace3.log: SolrException when authProvider.hasAccess() returns false due to # 2 above.

Also attached is a snippet from the KDC log - the full log bloats to 17 MB within a minute, full of messages like:

Apr 10 17:06:37 a0 krb5kdc[20427](info): TGS_REQ (1 etypes {23}) 10.0.0.1: ISSUE: authtime 1491818430, etypes {rep=23 tkt=23 ses=23}, solr/[e-mail] for sentry/[e-mail]

This is reproducible in two separate clusters with different environments:
CDH 5.10.1 and
CDH 5.8.0

Please let me know if I've left out any key information.

Issue Links

• relates to: SENTRY-296 Sentry Service Client does not allow for connection pooling