Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-8138

replace_address cannot find node to be replaced node after seed node restart

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Won't Fix
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal
    • Since Version:

      Description

      If a node failed and a cluster was restarted (which is common case on massive outages), replace_address fails with

      Caused by: java.lang.RuntimeException: Cannot replace_address /172.19.56.97 because it doesn't exist in gossip
      jvm 1    | 	at org.apache.cassandra.service.StorageService.prepareReplacementInfo(StorageService.java:472)
      jvm 1    | 	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:724)
      jvm 1    | 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:686)
      jvm 1    | 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:562)
      

      Although neccessary information is saved in system tables on seed nodes, it is not loaded to gossip on seed node, so a replacement node cannot get this info.

      Attached patch loads all information from system tables to gossip with generation 0 and fixes some bugs around this info on shadow gossip round.

        Attachments

          Activity

            People

            • Assignee:
              brandon.williams Brandon Williams Assign to me
              Reporter:
              m0nstermind Oleg Anastasyev
              Authors:
              Brandon Williams

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment