Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1338

Tablet stuck in RaftConfig change currently pending

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: consensus
    • Labels:
      None

      Description

      We've been adapting the consensus logs for a while and I think we can finally get to the bottom of this issue. I'm attaching the logs from the 3 nodes that participated in the same config for tablet eaa1877a2b3540cf8202aff844c6ca79.

      ITBLL is driving the load and eventually fails at 2016-02-15 14:53:12,005 trying to write to node-2 AKA a1081edd2ca24f6b9dcdd7e5000f95ec. The peer that gets stuck is node-5 AKA cdec7fdacbac4ad1b095275b3bdbbe5c, starting from this line:

      I0215 14:28:41.585695  2020 raft_consensus_state.cc:459] T eaa1877a2b3540cf8202aff844c6ca79 P cdec7fdacbac4ad1b095275b3bdbbe5c [term 69 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
      

      The chaos monkey running on this setup is dropping packets one node at time.

      I'll attach the logs in a moment.

        Attachments

        1. KUDU_TSERVER.node-2.internal.gz
          264 kB
          Jean-Daniel Cryans
        2. KUDU_TSERVER.node-3.internal.gz
          288 kB
          Jean-Daniel Cryans
        3. KUDU_TSERVER.node-5.internal.gz
          926 kB
          Jean-Daniel Cryans
        4. logs.tgz
          191 kB
          Todd Lipcon

          Activity

            People

            • Assignee:
              tlipcon Todd Lipcon
              Reporter:
              jdcryans Jean-Daniel Cryans
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: