Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7350

Decommissioning nodes borks the seed node - can't add additional nodes

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Low
    • Resolution: Cannot Reproduce
    • Fix Version/s: 2.0.9
    • Component/s: None
    • Labels:
    • Environment:

      Ubuntu using the auto-clustering AMI

    • Severity:
      Low

      Description

      1) Launch a 4 node cluster - I used the auto-clustering AMI (you get nodes 0-3)
      2) decommission that last 2 nodes (nodes , leaving a 2 node cluster)
      3) wipe the data directories from node 2
      4) bootstrap node2 - it won't join "unable to gossip with any seeds".

      If you bootstrap the node a second time, it will join. However if you try to bootstrap node 3, it will also fail.

      I discovered that bouncing the seed node fixes the problem. I think it cropped up in 2.0.7.

      Error:
      ERROR [main] 2014-06-03 21:52:46,649 CassandraDaemon.java (line 497) Exception encountered during startup
      java.lang.RuntimeException: Unable to gossip with any seeds
      at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1193)
      at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:447)
      at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:656)
      at org.apache.cassandra.service.StorageService.initServer(StorageService.java:612)
      at org.apache.cassandra.service.StorageService.initServer(StorageService.java:505)
      at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:362)
      at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:480)
      at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:569)
      ERROR [StorageServiceShutdownHook] 2014-06-03 21:52:46,741 CassandraDaemon.java (line 198) Exception in thread Thread[StorageServi

        Attachments

          Activity

            People

            • Assignee:
              shawn.kumar Shawn Kumar Assign to me
              Reporter:
              slowenthal Steven Lowenthal
              Authors:
              Shawn Kumar
              Tester:
              Shawn Kumar

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment