Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15295

Running into deadlock when do CommitLog initialization

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 4.0
    • Component/s: Local/Commit Log
    • Labels:
      None

      Description

      Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a long time.
      I used jstack to saw what happened. The main thread stuck in AbstractCommitLogSegmentManager.awaitAvailableSegment

      The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it was not actually running. 

      And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on java class initialization.

      This is a deadlock obviously. CommitLog waits for a CommitLogSegment when initializing. In this moment, the CommitLog class is not initialized and the main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a CommitLogSegment with exception and call CommitLog.handleCommitError(static method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog class is still initializing.

       

       

        Attachments

        1. image.png
          583 kB
          David Capwell
        2. jstack.log
          15 kB
          Zephyr Guo
        3. pstack.log
          34 kB
          Zephyr Guo
        4. screenshot-1.png
          865 kB
          Zephyr Guo
        5. screenshot-2.png
          224 kB
          Zephyr Guo
        6. screenshot-3.png
          569 kB
          Zephyr Guo

          Issue Links

            Activity

              People

              • Assignee:
                gzh1992n Zephyr Guo
                Reporter:
                gzh1992n Zephyr Guo
                Authors:
                Dinesh Joshi, Zephyr Guo
                Reviewers:
                Dinesh Joshi, Jordan West
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: