Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15295

Running into deadlock when do CommitLog initialization

    XMLWordPrintableJSON

    Details

      Description

      Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a long time.
      I used jstack to saw what happened. The main thread stuck in AbstractCommitLogSegmentManager.awaitAvailableSegment

      The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it was not actually running. 

      And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on java class initialization.

      This is a deadlock obviously. CommitLog waits for a CommitLogSegment when initializing. In this moment, the CommitLog class is not initialized and the main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a CommitLogSegment with exception and call CommitLog.handleCommitError(static method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog class is still initializing.

       

       

        Attachments

        1. screenshot-3.png
          569 kB
          Zephyr Guo
        2. screenshot-2.png
          224 kB
          Zephyr Guo
        3. screenshot-1.png
          865 kB
          Zephyr Guo
        4. pstack.log
          34 kB
          Zephyr Guo
        5. jstack.log
          15 kB
          Zephyr Guo
        6. image.png
          583 kB
          David Capwell

          Issue Links

            Activity

              People

              • Assignee:
                gzh1992n Zephyr Guo
                Reporter:
                gzh1992n Zephyr Guo
                Authors:
                Dinesh Joshi, Zephyr Guo
                Reviewers:
                Dinesh Joshi, Jordan West
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: