Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-4491

Deadlock happens to Zeppelin when deleting multiple paragraphs synchronously via RestAPI

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: Core
    • Labels:
    • Flags:
      Patch

      Description

      The zeppelin server is consistently getting deadlocked when delete multiple paragraphs which belong to different notes synchronously

      The RestApi called is below: 

      // zeppelin-server\src\main\java\org\apache\zeppelin\rest\NotebookRestApi.java
      /**
      
      Delete paragraph REST API
       * @param noteId ID of Note @return JSON with status.OK @throws IOException
       */
       @DELETE
       @Path("{noteId}/paragraph/{paragraphId}")
       @ZeppelinApi
       public Response deleteParagraph(@PathParam("noteId") String noteId,
       @PathParam("paragraphId") String paragraphId) throws IOException {
       LOG.info("delete paragraph {} {}", noteId, paragraphId);
      Note note = notebook.getNote(noteId);
       checkIfNoteIsNotNull(note);
       checkIfUserCanRead(noteId,
       "Insufficient privileges you cannot remove paragraph from this note");
      Paragraph p = note.getParagraph(paragraphId);
       checkIfParagraphIsNotNull(p);
      AuthenticationInfo subject = new AuthenticationInfo(SecurityUtils.getPrincipal());
       note.removeParagraph(SecurityUtils.getPrincipal(), paragraphId);
       note.persist(subject);
       notebookServer.broadcastNote(note);
      return new JsonResponse(Status.OK, "").build();
       }
      

       

      The relevant thread dump is below:
       

      Server compiler detected.
       JVM version is 25.212-b10
       Deadlock Detection:
      Found one Java-level deadlock:
       =============================
      "qtp1188392295-14051":
       waiting to lock Monitor@0x00007f3dc800a108 (Object@0x00000000c7706788, a java/util/LinkedList),
       which is held by "qtp1188392295-14087"
       "qtp1188392295-14087":
       waiting to lock Monitor@0x00007f3d10009278 (Object@0x00000000c78d3e78, a java/util/LinkedList),
       which is held by "qtp1188392295-14051"
      Found a total of 1 deadlock.
      

      The stacktrace information is below:
       

      "qtp1188392295-14087":
       [1] org.apache.zeppelin.notebook.Note.getParagraphs (Note.java:750)
       [2] org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime (Notebook.java:871)
       [3] org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNoteJobInfo (NotebookServer.java:554)
       [4] org.apache.zeppelin.socket.NotebookServer$NotebookInformationListener.onParagraphRemove (NotebookServer.java:2,180)
       [5] org.apache.zeppelin.notebook.Notebook.onParagraphRemove (Notebook.java:1,083)
       [6] org.apache.zeppelin.notebook.Note.removeParagraph (Note.java:446)
       [7] org.apache.zeppelin.rest.NotebookRestApi.deleteParagraph (NotebookRestApi.java:615)
       [8] sun.reflect.GeneratedMethodAccessor27.invoke (null)
       [9] sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
       [10] java.lang.reflect.Method.invoke (Method.java:498)
       [11] org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0 (ResourceMethodInvocationHandlerFactory.java:76)
       [12] org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$159.invoke (null)
       [13] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run (AbstractJavaResourceMethodDispatcher.java:148)
       [14] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke (AbstractJavaResourceMethodDispatcher.java:191)
       [15] org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch (JavaResourceMethodDispatcherProvider.java:200)
       [16] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch (AbstractJavaResourceMethodDispatcher.java:103)
       [17] org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke (ResourceMethodInvoker.java:493)
       [18] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply (ResourceMethodInvoker.java:415)
       [19] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply (ResourceMethodInvoker.java:104)
       [20] org.glassfish.jersey.server.ServerRuntime$1.run (ServerRuntime.java:277)
       [21] org.glassfish.jersey.internal.Errors$1.call (Errors.java:272)
       [22] org.glassfish.jersey.internal.Errors$1.call (Errors.java:268)
       [23] org.glassfish.jersey.internal.Errors.process (Errors.java:316)
       [24] org.glassfish.jersey.internal.Errors.process (Errors.java:298)
       [25] org.glassfish.jersey.internal.Errors.process (Errors.java:268)
       [26] org.glassfish.jersey.process.internal.RequestScope.runInScope (RequestScope.java:289)
       [27] org.glassfish.jersey.server.ServerRuntime.process (ServerRuntime.java:256)
       [28] org.glassfish.jersey.server.ApplicationHandler.handle (ApplicationHandler.java:703)
       [29] org.glassfish.jersey.servlet.WebComponent.serviceImpl (WebComponent.java:416)
       [30] org.glassfish.jersey.servlet.WebComponent.service (WebComponent.java:370)
       [31] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:389)
       [32] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:342)
       [33] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:229)
       [34] org.eclipse.jetty.servlet.ServletHolder.handle (ServletHolder.java:865)
       [35] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,655)
       [36] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter (ProxiedFilterChain.java:61)
       [37] org.apache.shiro.web.servlet.AdviceFilter.executeChain (AdviceFilter.java:108)
       [38] org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal (AdviceFilter.java:137)
       [39] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:125)
       [40] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter (ProxiedFilterChain.java:66)
       [41] org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain (AbstractShiroFilter.java:449)
       [42] org.apache.shiro.web.servlet.AbstractShiroFilter$1.call (AbstractShiroFilter.java:365)
       [43] org.apache.shiro.subject.support.SubjectCallable.doCall (SubjectCallable.java:90)
       [44] org.apache.shiro.subject.support.SubjectCallable.call (SubjectCallable.java:83)
       [45] org.apache.shiro.subject.support.DelegatingSubject.execute (DelegatingSubject.java:383)
       [46] org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal (AbstractShiroFilter.java:362)
       [47] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:125)
       [48] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,642)
       [49] org.apache.zeppelin.server.CorsFilter.doFilter (CorsFilter.java:72)
       [50] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,642)
       [51] org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:533)
       [52] org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:146)
       [53] org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:548)
       [54] org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)
       [55] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:257)
       [56] org.eclipse.jetty.server.session.SessionHandler.doHandle (SessionHandler.java:1,595)
       [57] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:255)
       [58] org.eclipse.jetty.server.handler.ContextHandler.doHandle (ContextHandler.java:1,317)
       [59] org.eclipse.jetty.server.handler.ScopedHandler.nextScope (ScopedHandler.java:203)
       [60] org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:473)
       [61] org.eclipse.jetty.server.session.SessionHandler.doScope (SessionHandler.java:1,564)
       [62] org.eclipse.jetty.server.handler.ScopedHandler.nextScope (ScopedHandler.java:201)
       [63] org.eclipse.jetty.server.handler.ContextHandler.doScope (ContextHandler.java:1,219)
       [64] org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:144)
       [65] org.eclipse.jetty.server.handler.ContextHandlerCollection.handle (ContextHandlerCollection.java:219)
       [66] org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)
       [67] org.eclipse.jetty.server.Server.handle (Server.java:531)
       [68] org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352)
       [69] org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:260)
       [70] org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:281)
       [71] org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102)
       [72] org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
       [73] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask (EatWhatYouKill.java:333)
       [74] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:310)
       [75] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce (EatWhatYouKill.java:168)
       [76] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:126)
       [77] org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:366)
       [78] org.eclipse.jetty.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:762)
       [79] org.eclipse.jetty.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:680)
       [80] java.lang.Thread.run (Thread.java:748)
      ------------------------------------------------------------------------------------------------
      "qtp1188392295-14051":
       [1] org.apache.zeppelin.notebook.Note.getParagraphs (Note.java:750)
       [2] org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime (Notebook.java:871)
       [3] org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNoteJobInfo (NotebookServer.java:554)
       [4] org.apache.zeppelin.socket.NotebookServer$NotebookInformationListener.onParagraphRemove (NotebookServer.java:2,180)
       [5] org.apache.zeppelin.notebook.Notebook.onParagraphRemove (Notebook.java:1,083)
       [6] org.apache.zeppelin.notebook.Note.removeParagraph (Note.java:446)
       [7] org.apache.zeppelin.rest.NotebookRestApi.deleteParagraph (NotebookRestApi.java:615)
       [8] sun.reflect.GeneratedMethodAccessor27.invoke (null)
       [9] sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
       [10] java.lang.reflect.Method.invoke (Method.java:498)
       [11] org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0 (ResourceMethodInvocationHandlerFactory.java:76)
       [12] org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$$Lambda$159.invoke (null)
       [13] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run (AbstractJavaResourceMethodDispatcher.java:148)
       [14] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke (AbstractJavaResourceMethodDispatcher.java:191)
       [15] org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch (JavaResourceMethodDispatcherProvider.java:200)
       [16] org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch (AbstractJavaResourceMethodDispatcher.java:103)
       [17] org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke (ResourceMethodInvoker.java:493)
       [18] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply (ResourceMethodInvoker.java:415)
       [19] org.glassfish.jersey.server.model.ResourceMethodInvoker.apply (ResourceMethodInvoker.java:104)
       [20] org.glassfish.jersey.server.ServerRuntime$1.run (ServerRuntime.java:277)
       [21] org.glassfish.jersey.internal.Errors$1.call (Errors.java:272)
       [22] org.glassfish.jersey.internal.Errors$1.call (Errors.java:268)
       [23] org.glassfish.jersey.internal.Errors.process (Errors.java:316)
       [24] org.glassfish.jersey.internal.Errors.process (Errors.java:298)
       [25] org.glassfish.jersey.internal.Errors.process (Errors.java:268)
       [26] org.glassfish.jersey.process.internal.RequestScope.runInScope (RequestScope.java:289)
       [27] org.glassfish.jersey.server.ServerRuntime.process (ServerRuntime.java:256)
       [28] org.glassfish.jersey.server.ApplicationHandler.handle (ApplicationHandler.java:703)
       [29] org.glassfish.jersey.servlet.WebComponent.serviceImpl (WebComponent.java:416)
       [30] org.glassfish.jersey.servlet.WebComponent.service (WebComponent.java:370)
       [31] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:389)
       [32] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:342)
       [33] org.glassfish.jersey.servlet.ServletContainer.service (ServletContainer.java:229)
       [34] org.eclipse.jetty.servlet.ServletHolder.handle (ServletHolder.java:865)
       [35] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,655)
       [36] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter (ProxiedFilterChain.java:61)
       [37] org.apache.shiro.web.servlet.AdviceFilter.executeChain (AdviceFilter.java:108)
       [38] org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal (AdviceFilter.java:137)
       [39] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:125)
       [40] org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter (ProxiedFilterChain.java:66)
       [41] org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain (AbstractShiroFilter.java:449)
       [42] org.apache.shiro.web.servlet.AbstractShiroFilter$1.call (AbstractShiroFilter.java:365)
       [43] org.apache.shiro.subject.support.SubjectCallable.doCall (SubjectCallable.java:90)
       [44] org.apache.shiro.subject.support.SubjectCallable.call (SubjectCallable.java:83)
       [45] org.apache.shiro.subject.support.DelegatingSubject.execute (DelegatingSubject.java:383)
       [46] org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal (AbstractShiroFilter.java:362)
       [47] org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter (OncePerRequestFilter.java:125)
       [48] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,642)
       [49] org.apache.zeppelin.server.CorsFilter.doFilter (CorsFilter.java:72)
       [50] org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1,642)
       [51] org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:533)
       [52] org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:146)
       [53] org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:548)
       [54] org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)
       [55] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:257)
       [56] org.eclipse.jetty.server.session.SessionHandler.doHandle (SessionHandler.java:1,595)
       [57] org.eclipse.jetty.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:255)
       [58] org.eclipse.jetty.server.handler.ContextHandler.doHandle (ContextHandler.java:1,317)
       [59] org.eclipse.jetty.server.handler.ScopedHandler.nextScope (ScopedHandler.java:203)
       [60] org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:473)
       [61] org.eclipse.jetty.server.session.SessionHandler.doScope (SessionHandler.java:1,564)
       [62] org.eclipse.jetty.server.handler.ScopedHandler.nextScope (ScopedHandler.java:201)
       [63] org.eclipse.jetty.server.handler.ContextHandler.doScope (ContextHandler.java:1,219)
       [64] org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:144)
       [65] org.eclipse.jetty.server.handler.ContextHandlerCollection.handle (ContextHandlerCollection.java:219)
       [66] org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)
       [67] org.eclipse.jetty.server.Server.handle (Server.java:531)
       [68] org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:352)
       [69] org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:260)
       [70] org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:281)
       [71] org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:102)
       [72] org.eclipse.jetty.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
       [73] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask (EatWhatYouKill.java:333)
       [74] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:310)
       [75] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce (EatWhatYouKill.java:168)
       [76] org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:126)
       [77] org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:366)
       [78] org.eclipse.jetty.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:762)
       [79] org.eclipse.jetty.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:680)
       [80] java.lang.Thread.run (Thread.java:748)
      

       
       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              MickelJS KylinJN
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: