[ACCUMULO-3695] Authentication check for system user incorrect for multiple nodes

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.7.0
• Component/s: master, monitor, trace, tserver
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 25/Mar/15 01:29
• Updated: 25/Mar/15 02:30
• Resolved: 25/Mar/15 02:30

Description

Was testing out a multiple node install of Accumulo with Kerberos via Ambari (post AMBARI-10163), and noticed that all but the one node could communicate out.

2015-03-25 01:08:19,252 [impl.SecurityOperationsImpl] DEBUG: Provided credentials did not match server's expected credentials. Expected org.apache.accumulo.server.security.SystemCredentials:accumulo/[e-mail]:org.apache.accumulo.server.security.SystemCredentials$SystemToken:<hidden> but got org.apache.accumulo.core.security.Credentials:accumulo/[e-mail]:org.apache.accumulo.server.security.SystemCredentials$SystemToken:<hidden>
2015-03-25 01:08:19,258 [tserver.TabletServer] WARN : Got loadTablet message from unauthenticatable user: accumulo/[e-mail]
2015-03-25 01:08:19,258 [tserver.TabletServer] ERROR: Got message from a service with a mismatched configuration. Please ensure a compatible configuration.
ThriftSecurityException(user:accumulo/[e-mail], code:BAD_CREDENTIALS)
        at org.apache.accumulo.server.security.SecurityOperation.authenticate(SecurityOperation.java:170)
        at org.apache.accumulo.server.security.AuditedSecurityOperation.authenticate(AuditedSecurityOperation.java:450)
        at org.apache.accumulo.server.security.SecurityOperation.canPerformSystemActions(SecurityOperation.java:437)
        at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.checkPermission(TabletServer.java:1385)
        at org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.loadTablet(TabletServer.java:1445)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.accumulo.core.trace.wrappers.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
        at org.apache.accumulo.server.rpc.RpcWrapper$1.invoke(RpcWrapper.java:47)
        at com.sun.proxy.$Proxy22.loadTablet(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invokeMethod(TCredentialsUpdatingInvocationHandler.java:154)
        at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:58)
        at com.sun.proxy.$Proxy22.loadTablet(Unknown Source)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$loadTablet.getResult(TabletClientService.java:2633)
        at org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$loadTablet.getResult(TabletClientService.java:2619)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:102)
        at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:745)

I think what's happening is that when it realizes that it's the System user, it does a equality check on the Credentials object. This will never be true for multiple nodes since the instance component (the hostname) of the principal will always be different.

Issue Links

• is related to: ACCUMULO-2815 Kerberos authentication for clients