Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-4345

New (bootstrapping) Supplant Healthy Nodes

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Normal
    • Resolution: Duplicate
    • None
    • None

    Description

      Copied a config from an existing node and fired up a new node, which happily inserted itself at token 0 of a running ring. The surprising and worrisome part is that EVERY node started throwing:

      java.lang.NullPointerException
      ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,948 Cassandra.java (line 3041) Internal error processing get_slice
      java.lang.NullPointerException
      ERROR [RPC-Thread:205427] 2012-06-14 19:16:31,949 Cassandra.java (line 3041) Internal error processing get_slice
      java.lang.NullPointerException
      ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,952 Cassandra.java (line 3041) Internal error processing get_slice
      java.lang.NullPointerException

      Resulting in:
      INFO [GossipStage:1] 2012-06-14 18:24:37,472 Gossiper.java (line 838) Node /192.168.88.48 is now part of the cluster
      INFO [GossipStage:1] 2012-06-14 18:24:37,473 Gossiper.java (line 804) InetAddress /192.168.88.48 is now UP
      INFO [GossipStage:1] 2012-06-14 18:24:37,473 StorageService.java (line 1008) Nodes /192.168.88.48 and /192.168.88.70 have the same token 0. /192.168.88
      .48 is the new owner
      WARN [GossipStage:1] 2012-06-14 18:24:37,474 TokenMetadata.java (line 135) Token 0 changing ownership from /192.168.88.70 to /192.168.88.48
      INFO [GossipStage:1] 2012-06-14 18:24:37,475 ColumnFamilyStore.java (line 705) Enqueuing flush of Memtable-LocationInfo@961917618(20/25 serialized/live
      bytes, 1 ops)
      INFO [FlushWriter:1272] 2012-06-14 18:24:37,475 Memtable.java (line 246) Writing Memtable-LocationInfo@961917618(20/25 serialized/live bytes, 1 ops)
      INFO [FlushWriter:1272] 2012-06-14 18:24:37,492 Memtable.java (line 283) Completed flushing /cass_ssd/system/LocationInfo-hc-23-Data.db (74 bytes)
      ERROR [RPC-Thread:200943] 2012-06-14 18:24:38,007 Cassandra.java (line 3041) Internal error processing get_slice
      java.lang.NullPointerException
      at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:104)
      at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122)
      at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:93)
      at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:100)
      at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:2002)
      at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1996)
      at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:604)
      at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:564)
      at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:128)
      at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:283)
      at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:365)
      at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:326)
      at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033)
      at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
      at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
      at org.apache.cassandra.thrift.CustomTHsHaServer$Invocation.run(CustomTHsHaServer.java:105)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)

      Attachments

        Activity

          People

            Unassigned Unassigned
            bcoverston Benjamin Coverston
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: