[CALCITE-1922] Work around Jetty issue where Kerberos v5 OID is disallowed for SPNEGO authentication

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: avatica-1.11.0
• Component/s: avatica
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 03/Aug/17 21:29
• Updated: 01/Apr/19 19:05
• Resolved: 04/Aug/17 02:14

Description

This appears to be another Jetty bug around SPNEGO. Huge thank you to kishore1729 for his help in debugging this issue. I could not have done it without his help.

Deploying the Avatica server behind a reverse-proxy, we observed that the server would deny the authentication requests from the client (whereas the client talking directly to Avatica was successful). Pardon the Phoenix classes instead of Avatica itself:

2017-08-03 19:09:29,440 WARN org.apache.phoenix.shaded.org.eclipse.jetty.security.SpnegoLoginService:
GSSException: No credential found for: 1.2.840.113554.1.2.2 usage: Accept
        at sun.security.jgss.GSSCredentialImpl.getElement(GSSCredentialImpl.java:600)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:317)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
        at org.apache.phoenix.shaded.org.eclipse.jetty.security.SpnegoLoginService.login(SpnegoLoginService.java:137)
        at org.apache.phoenix.shaded.org.eclipse.jetty.security.authentication.LoginAuthenticator.login(LoginAuthenticator.java:61)
        at org.apache.phoenix.shaded.org.eclipse.jetty.security.authentication.SpnegoAuthenticator.validateRequest(SpnegoAuthenticator.java:99)
        at org.apache.calcite.avatica.server.AvaticaSpnegoAuthenticator.validateRequest(AvaticaSpnegoAuthenticator.java:43)
        at org.apache.phoenix.shaded.org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:512)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
        at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:748)
2017-08-03 19:09:29,441 DEBUG org.apache.calcite.avatica.server.AvaticaJsonHandler: HTTP request from 10.0.0.63 is unauthenticated and authentication is required

Investigating this further, we found that Jetty's SpnegoLoginService was explicitly only allowing an OID of 1.3.6.1.5.5.2 instead of allowing both 1.3.6.1.5.5.2 for SPNEGO and 1.2.840.113554.1.2.2 for Kerberos v5 (e.g. See Presto's SpnegoFilter class: https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/server/security/SpnegoFilter.java#L113-L114). Best as I can tell, this is a limitation in Jetty to only allow the SPNEGO OID and not both.

We were able to observe that this wasn't a problem with the "stock" Avatica client that uses commons-httpclient because it sends both the kerberos v5 OID and spnego OID (each with data, of course).

We need to add a workaround to Avatica while we get this upstream in Jetty.

Issue Links

• is related to: CALCITE-2972 Upgrade jetty to 9.4.15.v20190215
• links to: GitHub Pull Request #15