[RANGER-1826] Import of bulk policies is causing OOM and Apparent Deadlock

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0, 1.0.0
• Fix Version/s: 1.0.0, 0.7.2
• Component/s: Ranger
• Assignee: Pradeep Agrawal
• Reporter: Deepak Sharma
• Created: 06/Oct/17 05:13
• Updated: 10/Oct/17 09:18
• Resolved: 10/Oct/17 09:18

Description

while importing bulk polices around 1000, OOM error is thrown:

2017-09-05 10:19:53,856 [http-bio-6080-exec-13] ERROR org.apache.ranger.rest.ServiceREST (ServiceREST.java:1603) - deletePolicy(282) failed
java.lang.OutOfMemoryError: Java heap space
2017-09-05 10:19:53,857 [http-bio-6080-exec-13] INFO  org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:63) - Request failed. loginId=admin, logMessage=Java heap space
2017-09-05 10:19:54,481 [http-bio-6080-exec-13] ERROR org.apache.ranger.rest.ServiceREST (ServiceREST.java:2117) - Error while importing policy from file!!
java.lang.OutOfMemoryError: GC overhead limit exceeded
2017-09-05 10:19:54,483 [http-bio-6080-exec-13] INFO  org.apache.ranger.common.RESTErrorUtil (RESTErrorUtil.java:63) - Request failed. loginId=admin, logMessage=GC overhead limit exceeded
javax.ws.rs.WebApplicationException
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:311)
        at org.apache.ranger.rest.ServiceREST.importPoliciesFromFile(ServiceREST.java:2125)
        at org.apache.ranger.rest.ServiceREST$$FastClassBySpringCGLIB$$92dab672.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:700)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64)

then Increased the heap size:
${JAVA_OPTS} -XX:MaxPermSize=256m -Xmx2048m -Xms1024m

then restarted the ranger service and imported the policies again:

it causes apparent deadlock

2017-09-05 13:38:22,966 [Timer-0] WARN  com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector (ThreadPoolAsynchronousRunner.java:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6726e60e -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!

and after sometime it starts giving OOM again:

2017-09-05 13:50:54,670 [http-bio-6080-exec-4] ERROR org.apache.ranger.rest.ServiceREST (ServiceREST.java:1603) - deletePolicy(256) failed
java.lang.OutOfMemoryError: Java heap space
2017-09-05 13:52:08,355 [zkCallback-2-thread-5-EventThread] WARN  org.apache.solr.common.cloud.ConnectionManager (ConnectionManager.java:183) - zkClient received AuthFailed
2017-09-05 13:52:20,662 [http-bio-6080-exec-4] INFO  ? (?:?) - Request failed. loginId=admin, logMessage=Java heap space
javax.ws.rs.WebApplicationException
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:56)
        at org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:311)
        at org.apache.ranger.rest.ServiceREST.deletePolicy(ServiceREST.java:1605)
        at org.apache.ranger.rest.ServiceREST.deletePoliciesProvidedInServiceMap(ServiceREST.java:224

Issue Links

• links to: Review Request Link-1