Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Cannot Reproduce
-
Impala 2.11.0
-
None
-
CDH 5.14, CDH 5.13, Centos 7, OpenLDAP
-
ghx-label-4
Description
Environment:
CDH 5.14.2 (upgraded from CDH 5.13.0 that was also affected), Sentry configured, Hadoop group mapping via Ldap (OpenLDAP). The cluster is NOT Kerberized.
Issue description:
There are some roles setup on Impala+Sentry:
CREATE ROLE dba; GRANT ALL ON SERVER server1 TO ROLE dba WITH GRANT OPTION; GRANT ROLE dba TO GROUP `gn:ldap:admin`; CREATE ROLE etl; GRANT ALL ON SERVER server1 TO ROLE etl; GRANT ROLE etl TO GROUP `gn:users:etl`; CREATE ROLE bi; GRANT ROLE bi TO GROUP `gn:users:bi`; grant select on database reporting to role bi;
Just after permissions grant was made, al works fine and users are assigned correct roles, which is confirmed by SHOW CURRENT ROLES on Impala side.
But after Impala restart, any user that is logged to Impala does not have any roles. So, SHOW CURRENT ROLES returns just nothing. And users do not have any permissions, even users who are in gn:ldap:admin (it is stated as sentry.service.admin.group in Sentry).
hdfs groups <username> returns correct groups for a user, so group mapping obviously work fine.
Also, I can see the correct roles for a user, if I log to Hue->Security-> Roles.
So that the issue seems to be somewhere between Impala and Sentry.
The workaround for this that I am currently using is to create another set of roles as dba1, etl1, bi1 and grant them to appropriate groups. Another workaround is to stop Sentry service, log in to Sentry database, drop all the tables, recreate Sentry schema and then create roles and grant permissions again.
At the same time, I see a lot of such logs in Sentry Server logs:
Access denied to impala org.apache.sentry.provider.db.SentryAccessDeniedException: Access denied to impala at org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.list_sentry_roles_by_group(SentryPolicyStoreProcessor.java:581) at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$list_sentry_roles_by_group.getResult(SentryPolicyService.java:1077) at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$list_sentry_roles_by_group.getResult(SentryPolicyService.java:1062) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:36) at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Also, a lot of these logs in Impala Catalog Server logs:
2:10:18.844 PM ERROR java:99 failed to execute listRoles Java exception follows: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sentry.org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95) at sentry.org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41) at com.sun.proxy.$Proxy19.listRoles(Unknown Source) at org.apache.impala.util.SentryPolicyService.listAllRoles(SentryPolicyService.java:393) at org.apache.impala.util.SentryProxy$PolicyReader.run(SentryProxy.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.sentry.provider.db.SentryAccessDeniedException: Access denied to impala. Server Stacktrace: org.apache.sentry.provider.db.SentryAccessDeniedException: Access denied to impala at org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.list_sentry_roles_by_group(SentryPolicyStoreProcessor.java:581) at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$list_sentry_roles_by_group.getResult(SentryPolicyService.java:1077) at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$list_sentry_roles_by_group.getResult(SentryPolicyService.java:1062) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:36) at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at org.apache.sentry.service.thrift.Status.throwIfNotOk(Status.java:113) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl.listRolesByGroupName(SentryPolicyServiceClientDefaultImpl.java:161) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl.listRoles(SentryPolicyServiceClientDefaultImpl.java:207) ... 15 more 2:10:18.844 PM ERROR java:176 Error refreshing Sentry policy: Java exception follows: org.apache.impala.catalog.AuthorizationException: User 'impala' does not have privileges to execute: LIST_ROLES at org.apache.impala.util.SentryPolicyService.listAllRoles(SentryPolicyService.java:395) at org.apache.impala.util.SentryProxy$PolicyReader.run(SentryProxy.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Please let me know what other information is needed.