Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-5736

NiFi can deadlock if one component reads FlowFile from Connection at the same time another component adds to it

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: None
    • Labels:
      None

      Description

      I ran into a condition that results in a NiFi processor deadlocking, with the following 3 deadlocked threads in the thread dump:

      "Timer-Driven Process Thread-9" Id=75 WAITING on java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@70edf082 ** DEADLOCKED THREAD **
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
      at org.apache.nifi.util.concurrency.DebugDisabledTimedLock.lock(DebugDisabledTimedLock.java:56)
      at org.apache.nifi.util.concurrency.TimedLock.lock(TimedLock.java:52)
      at org.apache.nifi.controller.queue.SwappablePriorityQueue.poll(SwappablePriorityQueue.java:469)
      at org.apache.nifi.controller.queue.StandardFlowFileQueue.poll(StandardFlowFileQueue.java:129)
      at org.apache.nifi.connectable.StandardConnection.poll(StandardConnection.java:354)
      at org.apache.nifi.controller.repository.StandardProcessSession.get(StandardProcessSession.java:1548)
      at org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.get(BatchingSessionFactory.java:87)
      at org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:241)
      at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
      at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
      at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
      at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Number of Locked Synchronizers: 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@53b3cd9b
      
      
      "Timer-Driven Process Thread-5" Id=66 BLOCKED on org.apache.nifi.controller.queue.SwappablePriorityQueue@32ae5c04 ** DEADLOCKED THREAD **
      at org.apache.nifi.controller.queue.SwappablePriorityQueue.getPriorities(SwappablePriorityQueue.java:107)
      at org.apache.nifi.controller.queue.SwappablePriorityQueue.writeSwapFilesIfNecessary(SwappablePriorityQueue.java:175)
      at org.apache.nifi.controller.queue.SwappablePriorityQueue.putAll(SwappablePriorityQueue.java:453)
      at org.apache.nifi.controller.queue.StandardFlowFileQueue.putAll(StandardFlowFileQueue.java:119)
      at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:442)
      at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:342)
      - waiting on org.apache.nifi.controller.repository.StandardProcessSession@69da1959
      at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
      at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
      at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
      at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Number of Locked Synchronizers: 2
      - java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@70edf082
      - java.util.concurrent.ThreadPoolExecutor$Worker@19adcb41
      
      
      
      "NiFi Web Server-25" Id=25 WAITING on java.util.concurrent.locks.ReentrantReadWriteLock$FairSync@70edf082 ** DEADLOCKED THREAD **
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
      at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
      at org.apache.nifi.util.concurrency.DebugDisabledTimedLock.lock(DebugDisabledTimedLock.java:56)
      at org.apache.nifi.util.concurrency.TimedLock.lock(TimedLock.java:52)
      at org.apache.nifi.controller.queue.SwappablePriorityQueue.getPriorities(SwappablePriorityQueue.java:107)
      - waiting on org.apache.nifi.controller.queue.SwappablePriorityQueue@32ae5c04
      at org.apache.nifi.controller.queue.StandardFlowFileQueue.getPriorities(StandardFlowFileQueue.java:97)
      at org.apache.nifi.web.api.dto.DtoFactory.createConnectionDto(DtoFactory.java:685)
      at org.apache.nifi.web.api.dto.DtoFactory.createFlowDto(DtoFactory.java:2173)
      at org.apache.nifi.web.api.dto.DtoFactory.createProcessGroupFlowDto(DtoFactory.java:2000)
      at org.apache.nifi.web.StandardNiFiServiceFacade.getProcessGroupFlow(StandardNiFiServiceFacade.java:3587)
      at org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke(<generated>)
      at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
      at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
      at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:84)
      at org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithReadLock(NiFiServiceFacadeLock.java:155)
      at org.apache.nifi.web.NiFiServiceFacadeLock.getLock(NiFiServiceFacadeLock.java:120)
      at sun.reflect.GeneratedMethodAccessor293.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)
      at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
      at org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$c416b321.getProcessGroupFlow(<generated>)
      at org.apache.nifi.web.api.FlowResource.getFlow(FlowResource.java:372)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
      at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$459/525141182.invoke(Unknown Source)
      at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
      at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
      at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
      at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
      at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
      at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
      at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
      at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
      at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
      at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
      at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
      at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
      at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
      at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
      at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
      at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
      at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
      at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
      at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
      at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
      at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
      at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
      at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
      at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
      at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
      at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
      at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:84)
      at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:58)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
      at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:84)
      at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:58)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
      at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:84)
      at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:58)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
      at org.apache.nifi.web.security.NiFiAuthenticationFilter.authenticate(NiFiAuthenticationFilter.java:84)
      at org.apache.nifi.web.security.NiFiAuthenticationFilter.doFilter(NiFiAuthenticationFilter.java:58)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
      at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
      at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
      at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
      at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
      at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
      at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.apache.nifi.web.filter.TimerFilter.doFilter(TimerFilter.java:51)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.apache.nifi.web.filter.ExceptionFilter.doFilter(ExceptionFilter.java:46)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
      at org.apache.nifi.web.security.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:47)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.apache.nifi.web.server.JettyServer$2.doFilter(JettyServer.java:1048)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
      at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
      at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
      at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:724)
      at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
      at org.eclipse.jetty.server.Server.handle(Server.java:531)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
      at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
      at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
      at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
      at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
      at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
      at java.lang.Thread.run(Thread.java:745)

        Attachments

          Activity

            People

            • Assignee:
              markap14 Mark Payne
              Reporter:
              markap14 Mark Payne
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: